Enhanced Bot Detection on Social Media Platforms Using RoBERTa and Multimodal Features

Okikiola D. Mojoyinola

1. Introduction

The proliferation of automated accounts (bots) on social media platforms presents significant challenges for platform integrity, information quality, and user experience. Bots can be used for various purposes, from benign automation to malicious activities such as spreading misinformation, manipulating public opinion, and artificially inflating engagement metrics. Detecting these automated accounts is crucial for maintaining the health of online discourse and ensuring the authenticity of social media interactions.

The increasing sophistication of online bots, particularly with the emergence of powerful Large Language Models (LLMs), presents a significant challenge. These advanced bots can generate remarkably human-like text and engage in complex interactions, making their detection more critical and difficult than ever. Incidents such as the undisclosed deployment of AI bots in online forums to influence user opinions [Yang, see references] highlight the urgency of this problem. Therefore, a primary aim of this project is to develop a robust detection mechanism capable of combating such issues. By leveraging the nuanced language understanding capabilities of advanced transformer models like RoBERTa, combined with insightful behavioral features, this work seeks to contribute to more reliable and adaptive bot detection systems that can evolve alongside the ever-improving capabilities of automated entities.

Traditional bot detection approaches have relied on rule-based systems or classical machine learning models using handcrafted features. While these methods have shown some success, they often struggle to adapt to the evolving nature of bot behavior and can be easily circumvented by sophisticated bots. More recent approaches have leveraged deep learning techniques, utilizing the power of models like transformers to understand complex patterns in data.

In this paper, we present a comprehensive bot detection system built upon these advancements. The core of our system is:

1. roberta-large based classification: We leverage the powerful language understanding capabilities of roberta-large, a variant of BERT, to extract rich semantic representations from tweet text, enabling more accurate detection of bot-generated content.
2. Multimodal Feature Integration: We combine the textual embeddings from roberta-large with numerical (e.g., follower counts, account age) and boolean (e.g., verified status) features extracted from user profiles to provide a more holistic view for classification.

Our system is designed to be a robust, multi-faceted approach to bot detection that can be deployed and adapted for real-world environments.

2. Related Work

Bot detection on social media has been an active area of research for over a decade. Early approaches focused on rule-based systems and manual feature engineering, while more recent work has increasingly leveraged machine learning and deep learning techniques.

2.1 Feature-Based Approaches

Numerous studies have explored the use of various features for bot detection. These features can be broadly categorized into:

• User-based features: Profile information, account age, verification status
• Content-based features: Text characteristics, sentiment, topic distribution
• Behavioral features: Posting patterns, engagement metrics, temporal activity
• Network-based features: Follower/following relationships, interaction patterns

For instance, Varol et al. (2017) developed the BotOrNot (later renamed Botometer) system, which uses over 1,000 features across these categories to classify Twitter accounts. Similarly, Yang et al. (2020) proposed a system that combines user, content, and temporal features for improved detection accuracy.

2.2 Deep Learning Approaches

More recent work has explored the use of deep learning techniques for bot detection. Kudugunta and Ferrara (2018) proposed an LSTM-based approach that combines text and metadata features. Wei and Nguyen (2019) developed a CNN-based model for detecting social bots based on tweet content. Feng et al. (2021) explored the use of transformer-based models for bot detection, showing promising results compared to traditional approaches. Our work aligns with this trajectory, specifically focusing on the roberta-large architecture and its integration with other feature types.

3. Methodology

Our bot detection system consists of several key components: data processing, feature handling, and model architecture. This section describes each component in detail, reflecting the implementation in the bot_detection_project codebase.

3.1 Data Processing (src/data_loader.py)

The system processes labeled Twitter data, used for supervised training. The TwitterDataProcessor class handles preprocessing:

• Text Preprocessing: Raw text (e.g., tweets) is cleaned by:
  • Replacing URLs with a special <url> token.
  • Replacing user mentions (@username) with <user>.
  • Replacing hashtags (#topic) with <hashtag> topic.
  • Normalizing whitespace.
  The cleaned text is then tokenized using the AutoTokenizer corresponding to roberta-large. Input sequences are padded or truncated to MAX_LEN (default 128 tokens).
• Numerical Feature Processing: Numerical features (e.g., followers_count, account_age_days) are scaled using StandardScaler. The scaler is fitted on the training data and applied to validation and test sets. Missing values are imputed with 0.
• Boolean Feature Processing: Boolean features (e.g., verified) are converted to floating-point numbers (0.0 or 1.0).
• Data Splitting: The dataset is split into training, validation, and test sets.
• Dataset and DataLoader: Custom PyTorch BotDataset and DataLoader classes manage data batching.

3.2 Feature Handling

Our system leverages multimodal features:

• Textual Features: These are implicitly handled by the roberta-large model, which processes the tokenized tweet text to generate contextual embeddings.
• Numerical and Boolean Features: Explicitly defined in config.py (NUMERICAL_FEATURES, BOOLEAN_FEATURES). These include:
  • Account metrics: e.g., follower count, following count, tweet count.
  • Verification status: Whether the account is verified.
  • Temporal features: e.g., account_age_days derived from account creation dates.
  These features are concatenated with the text embeddings from roberta-large before being passed to the classification head.

3.3 Model Architecture (src/models/bert_model.py)

The core of our system is the BERTBotDetector model:

1. Base Model: A roberta-large model (specified by MODEL_NAME in src/config.py) is loaded. To leverage its strong pre-trained knowledge while adapting to the specific task, a partial freezing strategy is employed by default: the RoBERTa embeddings and the initial 20 out of 24 encoder layers are frozen. The top 4 encoder layers and the model's pooler layer remain trainable, allowing for task-specific fine-tuning. The FREEZE_BERT_LAYERS configuration in config.py can be used to adjust this behavior for other BERT-based models if needed, though the RoBERTa default is handled directly in the trainer.
2. Input: The model takes tokenized text (input_ids, attention_mask), processed numerical features, and processed boolean features.
3. Text Representation: The [CLS] token output from roberta-large serves as the primary text embedding.
4. Feature Integration and Classification Head:
   • The text embedding is concatenated with the numerical and boolean feature vectors.
   • This combined vector passes through:
     • torch.nn.LayerNorm for normalization.
     • A fully connected layer (torch.nn.Linear) to a hidden dimension (HIDDEN_SIZE, default 128).
     • torch.nn.ReLU activation.
     • torch.nn.Dropout (DROPOUT_RATE, default 0.3).
     • A final torch.nn.Linear layer to 2 output units (for bot/non-bot logits).

The model is trained using torch.nn.CrossEntropyLoss, with optional class weighting (USE_CLASS_WEIGHTS) to handle class imbalance.

4. Implementation (pipeline.py, src/trainer.py)

4.1 System Components

Our bot detection system is implemented as a modular Python project:

• src/data_loader.py: Handles data loading, preprocessing, and Dataset/DataLoader creation.
• src/config.py: Centralized configuration for paths, model parameters, and hyperparameters.
• src/models/bert_model.py: Defines the roberta-large based model architecture. Alternative architectures like LSTM and Logistic Regression are also present as stubs.
• src/trainer.py: Implements the supervised training and validation logic, including early stopping and loss functions.
• src/evaluator.py: Manages model evaluation on the test set.
• pipeline.py: Main script to orchestrate the data processing, training, and evaluation pipeline.
• main_inference.py: Script for making predictions on new data using a trained model.

4.2 Training Pipeline

The training pipeline executed by pipeline.py consists of:

1. Configuration Loading: Parameters are loaded from src/config.py.
2. Data Loading and Preprocessing: Utilizes TwitterDataProcessor and BotDataset.
3. Data Splitting: Divides data into train, validation, and test sets.
4. Model Initialization: Creates and initializes the BERTBotDetector model.
5. Optimizer and Scheduler Setup: Configures AdamW optimizer with differential learning rates (BERT_LR for RoBERTa layers, LEARNING_RATE for the head) and a learning rate scheduler (get_linear_schedule_with_warmup or ReduceLROnPlateau).
6. Supervised Training: The ModelTrainer executes the training loop, including gradient accumulation (GRAD_ACCUMULATION_STEPS) and gradient clipping (GRAD_CLIP).
7. Validation and Early Stopping: Monitors validation loss per epoch and saves the best model. Early stopping (EARLY_STOPPING_PATIENCE) prevents overfitting.
8. Model Evaluation: After training, the best model is evaluated on the test set using ModelEvaluator.

4.3 Key Hyperparameters (Defaults from src/config.py)

• BERT variant: roberta-large (MODEL_NAME)
• Maximum sequence length: 128 tokens (MAX_LEN)
• Batch size: 8 (BATCH_SIZE)
• Learning rate (head): 5e-4 (LEARNING_RATE)
• Learning rate (RoBERTa): 2e-5 (BERT_LR)
• Training epochs: 20 (EPOCHS) (with early stopping)
• Dropout rate: 0.3 (DROPOUT_RATE)
• Warmup steps: 0 (WARMUP_STEPS) (by default for linear scheduler)

These hyperparameters can be adjusted in src/config.py.

5. Evaluation (src/evaluator.py)

5.1 Evaluation Metrics

The system's performance is evaluated using standard classification metrics, calculated by the ModelEvaluator:

• Accuracy: Overall classification accuracy.
• Precision: Proportion of true positives among positive predictions (calculated per class, and averaged - macro, micro, weighted).
• Recall: Proportion of true positives identified (calculated per class, and averaged).
• F1 Score: Harmonic mean of precision and recall (calculated per class, and averaged).
• ROC AUC: Area under the Receiver Operating Characteristic curve.

A classification report and confusion matrix are also generated.

Initial evaluation of a model checkpoint from epoch 10 (out of 20 planned training epochs) on sample data yielded the following metrics:

• Accuracy: 0.67
• Precision: 0.62
• Recall: 0.63
• F1 Score: 0.60
• ROC AUC: 0.64

While these early results indicated potential, comprehensive training and more extensive evaluation were unfortunately limited by available computational resources, specifically the constraints of the Google Colab free tier.

6. Limitations and Future Work

While the model demonstrates promising capabilities, its performance and generalizability are subject to certain limitations, primarily related to the dataset:

• Dataset Characteristics: The primary limitation is the nature of the dataset used for training and evaluation. The model's effectiveness is highly dependent on the size, diversity, and specific characteristics of this data. A dataset that doesn't fully represent the breadth of bot behaviors or the nuances of interactions on different platforms may limit the model's real-world applicability. Expanding the dataset to include a wider variety of bot types, languages, and online environments is crucial for enhancing robustness.
• Computational Constraints: Training advanced models like roberta-large optimally requires substantial computational resources. The development and training conducted for this paper were subject to resource limitations (e.g., Google Colab free tier), which constrained the number of epochs and the scope of hyperparameter exploration.
• Evolving Bot Tactics: The landscape of bot development is dynamic, with new techniques emerging continuously. A model trained on historical data may see its performance degrade as bot strategies evolve.

Future work will prioritize acquiring and incorporating more diverse and extensive datasets. Additionally, we plan to explore more advanced feature engineering and investigate adaptive learning techniques to counteract the evolving nature of bots and improve the model's long-term efficacy.

7. Conclusion

In this paper, we presented a bot detection system that leverages the roberta-large transformer model integrated with multimodal (text, numerical, boolean) user features. The system employs a supervised learning paradigm with a robust training and evaluation pipeline, emphasizing configurability and modularity. By combining advanced NLP capabilities with user profile characteristics, the approach aims to provide effective and adaptable detection of automated accounts on social media platforms.

While the system demonstrates a strong foundational methodology, ongoing research and development are necessary to address challenges such as evolving bot tactics and computational demands. The described framework serves as a solid starting point for further enhancements and practical applications in mitigating the impact of malicious bots.

8. References