Smart contracts are immutable programs deployed on blockchain platforms, where vulnerabilities can translate directly into financial loss. Among the techniques proposed for their detection, recent work has relied increasingly on Large Language Models (LLMs) guided by carefully crafted prompts. However, the evidence accumulated so far rests almost entirely on the dense Transformer architecture behind the GPT line and on model snapshots already scheduled for deprecation, leaving open the question of whether the reported behaviors reflect a general property of LLMs or an artifact of that specific design. This paper presents an extended replication of the empirical study by Chen et al. on LLM-based smart contract vulnerability detection. We replicate the original protocol on GPT-4 as a baseline, followed by a differentiated replication, executing the protocol on representative models based on Mixture-of-Experts and RNN-like designs, together with an attempted preliminary evaluation of State Space Models. The exact replication confirms the findings of Chen et al., including the high-recall, low-precision detection pattern they report on GPT-4. Nevertheless, the differentiated replication shows that this pattern does not transfer uniformly across the evaluated models. Those from the same architectural family may also exhibit different behaviors, indicating that architectural family alone is insufficient to predict the resulting detection profile. We observe the same qualitative dense-Transformer profile in both GPT-4 and Gemma-3, despite a gap of roughly two orders of magnitude in parameter counts, whereas two Mixture-of-Experts models from different lineages exhibit qualitatively divergent behaviors, indicating that architectural family alone is insufficient to predict the detection profile.
Smart contract vulnerabilities pose significant security risks to the Ethereum ecosystem, driving the development of automated tools for detection and mitigation. Smart contracts are written in Solidity, a rapidly evolving programming language that continues to add features and improvements to enhance smart contract security. New versions of Solidity change the compilation process, potentially affecting how tools interpret and analyze smart contract code. We use Solidity as a case study to investigate a broader problem in software analysis: whether tool effectiveness remains stable as the underlying programming language and execution environment evolve. In this continuously evolving landscape, we investigated the compatibility of detection tools with different Solidity versions. More specifically, we analyzed detection tools by empirically assessing (i) their crash resilience, (ii) their detection effectiveness, and (iii) their execution time across different versions of Solidity. We collected 2,500 real-world smart contracts to construct a balanced wild dataset and created a separate injected-vulnerability dataset comprising 270 contracts. We extended SmartBugs, a framework that enables the execution of vulnerability detection tools for Solidity, adding 11 new tools. Results indicate that most tools are crash-resilient to version 0.4. However, 20% of the evaluated tools have been updated to remain compatible. Tool performance varies across Solidity versions and is affected by Solidity changes; execution time is higher in version 0.4.x, which was used for evaluation. We present a list of Solidity changes that affect tool performance and offer actionable takeaways for both researchers and practitioners.