{"id":407,"date":"2026-09-20T18:12:00","date_gmt":"2026-09-20T18:12:00","guid":{"rendered":"https:\/\/agentsafelabs.com\/blog\/?p=407"},"modified":"2026-09-20T18:18:03","modified_gmt":"2026-09-20T18:18:03","slug":"does-your-agent-framework-choice-actually-matter-for-security-we-ran-9360-trials-to-find-out","status":"publish","type":"post","link":"https:\/\/agentsafelabs.com\/blog\/does-your-agent-framework-choice-actually-matter-for-security-we-ran-9360-trials-to-find-out\/","title":{"rendered":"Does Your Agent Framework Choice Actually Matter for Security? We Ran 9,360 Trials to Find Out."},"content":{"rendered":"\n<p class=\"wp-block-paragraph\"><em>A controlled, payload-verified evaluation of seven agentic frameworks says: mostly no \u2014 with one small, honest exception.<\/em><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">If you&#8217;re building a tool-using LLM agent today, you&#8217;re choosing among a genuinely crowded field: LangChain, CrewAI, AutoGen, LlamaIndex, the OpenAI Agents SDK, Google&#8217;s Agent Development Kit, Semantic Kernel, or just talking to the model&#8217;s API directly and rolling your own orchestration. That&#8217;s a real decision with real engineering tradeoffs \u2014 but does it also change how <em>safe<\/em> your agent is against adversarial input?<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">We built AgentPort-Bench to answer that question properly, and the honest answer is more interesting than a simple yes or no.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">The confound nobody controls for<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Here&#8217;s the problem with most cross-framework security comparisons: when you run &#8220;the same attack&#8221; through two different frameworks, you&#8217;re not actually holding the attack constant. Every framework templates prompts differently, structures message roles differently, and serializes tool schemas differently. If Framework A shows a different attack-success rate than Framework B, you genuinely don&#8217;t know whether that&#8217;s because the <em>framework<\/em> changed the model&#8217;s behavior, or because the <em>adapter<\/em> quietly rephrased the attack on its way to the model.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">We found out how real this risk is the hard way. During our own data collection, we discovered CrewAI&#8217;s default agent construction didn&#8217;t use our shared baseline system prompt at all \u2014 it built one from separate role\/goal\/backstory fields, and the resulting text included the word &#8220;safely&#8221; that appeared nowhere in any other condition&#8217;s prompt. That&#8217;s not a subtle difference. Fixing it and re-collecting the affected trials measurably shifted the results (a statistically significant swing in the pass rate). If we hadn&#8217;t verified payload identity byte-for-byte across every single condition, we would have reported a &#8220;framework effect&#8221; that was actually an adapter bug.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">That&#8217;s the whole reason this evaluation exists: <strong>you can&#8217;t ask whether frameworks differ in safety impact until you&#8217;ve made sure they&#8217;re actually receiving the same attack.<\/strong><\/p>\n\n\n\n<h2 class=\"wp-block-heading\">The actual experiment<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Once payload identity was verified, we ran a controlled comparison across:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Eight execution conditions<\/strong>: a direct-API baseline plus seven agentic frameworks<\/li>\n\n\n\n<li><strong>Six models<\/strong> spanning three providers and two capability tiers<\/li>\n\n\n\n<li><strong>Five attack families<\/strong> mapped onto OWASP&#8217;s Agentic Security Initiative threat taxonomy<\/li>\n\n\n\n<li><strong>9,360 total trials<\/strong>, combining an original six-condition study with a later two-framework extension into one unified dataset \u2014 analyzed jointly here for the first time<\/li>\n<\/ul>\n\n\n\n<h2 class=\"wp-block-heading\">What actually explains the outcome<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Attack category dominates. Which model you&#8217;re using matters, but less. Which framework you&#8217;re using barely registers at all \u2014 about two orders of magnitude smaller an effect than attack category, and about one order of magnitude smaller than model choice.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">That much, a classical statistical test could already tell us. But a non-significant result only tells you &#8220;we couldn&#8217;t detect a difference&#8221; \u2014 not &#8220;the difference is small enough not to matter in practice.&#8221; Those are different claims, and conflating them is a common way papers overstate a null result.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">So we didn&#8217;t stop at the classical test. We also ran a formal equivalence test \u2014 the kind of statistical tool designed specifically to answer &#8220;is this effect small enough to call practically negligible,&#8221; with the acceptable margin decided <em>before<\/em> looking at the results, not after. Every single one of the 28 possible pairwise comparisons between our eight conditions came back equivalent under that pre-specified margin.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">That&#8217;s a genuinely stronger claim than &#8220;we found no significant difference,&#8221; and we think it&#8217;s the more useful one for anyone actually deciding which framework to build on.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">The one honest exception<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">We could have stopped there and called it a clean story: framework choice doesn&#8217;t matter, full stop. We didn&#8217;t, because the data doesn&#8217;t quite support that.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">CrewAI \u2014 even <em>after<\/em> fixing the system-prompt bug described above \u2014 retains a small, statistically real residual effect. It&#8217;s not large: it clears the equivalence bar we set, and it&#8217;s the single closest pairwise comparison to that bar, not the only one that failed it. It shows up uniformly against every other condition, old and new alike, not specifically against the two frameworks we added later. Our best guess \u2014 genuinely a guess, not a tested finding \u2014 is that CrewAI&#8217;s role\/goal\/backstory prompt structure can&#8217;t be delivered as a single message the way every other condition can, even once you match its <em>content<\/em>. We flag that as the natural next question, not as something we&#8217;ve confirmed.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">We think reporting this precisely \u2014 real, small, specific, not inflated into &#8220;CrewAI is unsafe&#8221; and not smoothed away into &#8220;no effect anywhere&#8221; \u2014 is more useful than either a falsely tidy or a falsely alarming headline.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Two things you&#8217;d only find if you went looking<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Along the way we ran into two practical gotchas worth knowing if you&#8217;re building on these frameworks yourself:<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong><em>A fully deterministic reasoning-token-exhaustion failure<\/em><\/strong><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">One frontier model, on one specific prompt, silently consumed its entire token budget on invisible internal reasoning and returned nothing \u2014 6 out of 6 times, across two different frameworks. If your evaluation harness treats an empty response as a simple failure-to-respond rather than checking whether the model just burned its whole budget thinking, you&#8217;ll misdiagnose this.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\"><em>A token-accounting trap<\/em><\/h4>\n\n\n\n<p class=\"wp-block-paragraph\">Google ADK and Semantic Kernel report reasoning-token usage under genuinely different conventions for the same underlying model \u2014 one folds reasoning tokens into the total output count, the other reports them separately. Compare the two frameworks&#8217; own usage numbers directly, without correcting for this, and you&#8217;ll manufacture a cost difference that doesn&#8217;t exist.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">The bottom line<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">If you&#8217;re choosing an agent framework based on security concerns alone, this data says: don&#8217;t. Put your effort into attack-surface coverage and model selection instead \u2014 that&#8217;s where the real variance lives. Framework choice is, for the eight conditions and five attack families we tested, close enough to a non-issue that we can say so with a formal statistical test behind it, not just a shrug.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">The one exception is small enough that it shouldn&#8217;t change your framework decision on its own \u2014 but it&#8217;s real enough that if you&#8217;re running CrewAI in production, it&#8217;s worth knowing about.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Full methodology, the complete 9,360-trial dataset provenance, and the statistical models are documented in our paper, &#8220;AgentPort-Bench: A Controlled Seven-Framework Evaluation of Agentic AI Security Portability,&#8221; currently under review. <\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><em>The evaluation framework is open source at <a href=\"https:\/\/github.com\/AgentSafeLabs\/safelabs-eval\">github.com\/AgentSafeLabs\/safelabs-eval<\/a>, and the AgentPort-Bench contributor submission repository is live at <a href=\"https:\/\/github.com\/AgentSafeLabs\/agentport-bench-results\">github.com\/AgentSafeLabs\/agentport-bench-results<\/a> if you&#8217;d like to submit your own framework&#8217;s results.<\/em><\/p>\n","protected":false},"excerpt":{"rendered":"<p>A controlled, payload-verified evaluation of seven agentic frameworks says: mostly no \u2014 with one small, honest exception. If you&#8217;re building a tool-using LLM agent today, you&#8217;re choosing among a genuinely crowded field: LangChain, CrewAI, AutoGen, LlamaIndex, the OpenAI Agents SDK, Google&#8217;s Agent Development Kit, Semantic Kernel, or just talking to the model&#8217;s API directly and [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":411,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[7,10,19,18],"class_list":["post-407","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-blog","tag-agentic-system","tag-owasp-agentic-security","tag-payload-verified-evaluation","tag-seven-agentic-frameworks"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v28.5 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>Does Your Agent Framework Choice Actually Matter for Security? We Ran 9,360 Trials to Find Out. - Agentsafelabs<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/agentsafelabs.com\/blog\/does-your-agent-framework-choice-actually-matter-for-security-we-ran-9360-trials-to-find-out\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Does Your Agent Framework Choice Actually Matter for Security? We Ran 9,360 Trials to Find Out. - Agentsafelabs\" \/>\n<meta property=\"og:description\" content=\"A controlled, payload-verified evaluation of seven agentic frameworks says: mostly no \u2014 with one small, honest exception. If you&#8217;re building a tool-using LLM agent today, you&#8217;re choosing among a genuinely crowded field: LangChain, CrewAI, AutoGen, LlamaIndex, the OpenAI Agents SDK, Google&#8217;s Agent Development Kit, Semantic Kernel, or just talking to the model&#8217;s API directly and [&hellip;]\" \/>\n<meta property=\"og:url\" content=\"https:\/\/agentsafelabs.com\/blog\/does-your-agent-framework-choice-actually-matter-for-security-we-ran-9360-trials-to-find-out\/\" \/>\n<meta property=\"og:site_name\" content=\"Agentsafelabs\" \/>\n<meta property=\"article:published_time\" content=\"2026-09-20T18:12:00+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2026-09-20T18:18:03+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/agentsafelabs.com\/blog\/wp-content\/uploads\/2026\/09\/Does-Your-Agent-Framework-Choice-Actually-Matter-for-Security-We-Ran-9360-Trials-to-Find-Out.png\" \/>\n\t<meta property=\"og:image:width\" content=\"1672\" \/>\n\t<meta property=\"og:image:height\" content=\"941\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"author\" content=\"Waqar Javed\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Waqar Javed\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"5 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\\\/\\\/agentsafelabs.com\\\/blog\\\/does-your-agent-framework-choice-actually-matter-for-security-we-ran-9360-trials-to-find-out\\\/#article\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/agentsafelabs.com\\\/blog\\\/does-your-agent-framework-choice-actually-matter-for-security-we-ran-9360-trials-to-find-out\\\/\"},\"author\":{\"name\":\"Waqar Javed\",\"@id\":\"https:\\\/\\\/agentsafelabs.com\\\/blog\\\/#\\\/schema\\\/person\\\/76dedaeec309dfebad90b82e70cd80d9\"},\"headline\":\"Does Your Agent Framework Choice Actually Matter for Security? We Ran 9,360 Trials to Find Out.\",\"datePublished\":\"2026-09-20T18:12:00+00:00\",\"dateModified\":\"2026-09-20T18:18:03+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\\\/\\\/agentsafelabs.com\\\/blog\\\/does-your-agent-framework-choice-actually-matter-for-security-we-ran-9360-trials-to-find-out\\\/\"},\"wordCount\":1108,\"commentCount\":0,\"image\":{\"@id\":\"https:\\\/\\\/agentsafelabs.com\\\/blog\\\/does-your-agent-framework-choice-actually-matter-for-security-we-ran-9360-trials-to-find-out\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/agentsafelabs.com\\\/blog\\\/wp-content\\\/uploads\\\/2026\\\/09\\\/Does-Your-Agent-Framework-Choice-Actually-Matter-for-Security-We-Ran-9360-Trials-to-Find-Out.png\",\"keywords\":[\"Agentic System\",\"OWASP Agentic Security\",\"payload-verified evaluation\",\"seven agentic frameworks\"],\"articleSection\":[\"Blog\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\\\/\\\/agentsafelabs.com\\\/blog\\\/does-your-agent-framework-choice-actually-matter-for-security-we-ran-9360-trials-to-find-out\\\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/agentsafelabs.com\\\/blog\\\/does-your-agent-framework-choice-actually-matter-for-security-we-ran-9360-trials-to-find-out\\\/\",\"url\":\"https:\\\/\\\/agentsafelabs.com\\\/blog\\\/does-your-agent-framework-choice-actually-matter-for-security-we-ran-9360-trials-to-find-out\\\/\",\"name\":\"Does Your Agent Framework Choice Actually Matter for Security? We Ran 9,360 Trials to Find Out. - Agentsafelabs\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/agentsafelabs.com\\\/blog\\\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\\\/\\\/agentsafelabs.com\\\/blog\\\/does-your-agent-framework-choice-actually-matter-for-security-we-ran-9360-trials-to-find-out\\\/#primaryimage\"},\"image\":{\"@id\":\"https:\\\/\\\/agentsafelabs.com\\\/blog\\\/does-your-agent-framework-choice-actually-matter-for-security-we-ran-9360-trials-to-find-out\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/agentsafelabs.com\\\/blog\\\/wp-content\\\/uploads\\\/2026\\\/09\\\/Does-Your-Agent-Framework-Choice-Actually-Matter-for-Security-We-Ran-9360-Trials-to-Find-Out.png\",\"datePublished\":\"2026-09-20T18:12:00+00:00\",\"dateModified\":\"2026-09-20T18:18:03+00:00\",\"author\":{\"@id\":\"https:\\\/\\\/agentsafelabs.com\\\/blog\\\/#\\\/schema\\\/person\\\/76dedaeec309dfebad90b82e70cd80d9\"},\"breadcrumb\":{\"@id\":\"https:\\\/\\\/agentsafelabs.com\\\/blog\\\/does-your-agent-framework-choice-actually-matter-for-security-we-ran-9360-trials-to-find-out\\\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/agentsafelabs.com\\\/blog\\\/does-your-agent-framework-choice-actually-matter-for-security-we-ran-9360-trials-to-find-out\\\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/agentsafelabs.com\\\/blog\\\/does-your-agent-framework-choice-actually-matter-for-security-we-ran-9360-trials-to-find-out\\\/#primaryimage\",\"url\":\"https:\\\/\\\/agentsafelabs.com\\\/blog\\\/wp-content\\\/uploads\\\/2026\\\/09\\\/Does-Your-Agent-Framework-Choice-Actually-Matter-for-Security-We-Ran-9360-Trials-to-Find-Out.png\",\"contentUrl\":\"https:\\\/\\\/agentsafelabs.com\\\/blog\\\/wp-content\\\/uploads\\\/2026\\\/09\\\/Does-Your-Agent-Framework-Choice-Actually-Matter-for-Security-We-Ran-9360-Trials-to-Find-Out.png\",\"width\":1672,\"height\":941,\"caption\":\"Does Your Agent Framework Choice Actually Matter for Security? We Ran 9,360 Trials to Find Out.\"},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/agentsafelabs.com\\\/blog\\\/does-your-agent-framework-choice-actually-matter-for-security-we-ran-9360-trials-to-find-out\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/agentsafelabs.com\\\/blog\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Does Your Agent Framework Choice Actually Matter for Security? We Ran 9,360 Trials to Find Out.\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/agentsafelabs.com\\\/blog\\\/#website\",\"url\":\"https:\\\/\\\/agentsafelabs.com\\\/blog\\\/\",\"name\":\"Agentsafelabs\",\"description\":\"\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/agentsafelabs.com\\\/blog\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\\\/\\\/agentsafelabs.com\\\/blog\\\/#\\\/schema\\\/person\\\/76dedaeec309dfebad90b82e70cd80d9\",\"name\":\"Waqar Javed\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/fbceb86c7be592091a2c62a020d43ea3b96a22bf32e19f062870d50e5bbc22b7?s=96&d=mm&r=g\",\"url\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/fbceb86c7be592091a2c62a020d43ea3b96a22bf32e19f062870d50e5bbc22b7?s=96&d=mm&r=g\",\"contentUrl\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/fbceb86c7be592091a2c62a020d43ea3b96a22bf32e19f062870d50e5bbc22b7?s=96&d=mm&r=g\",\"caption\":\"Waqar Javed\"},\"description\":\"Waqar Javed is the founder of AgentSafeLabs and a researcher focused on empirical evaluation of AI agent security. safelabs-eval is Apache 2.0 licensed and available on PyPI and GitHub.\",\"sameAs\":[\"https:\\\/\\\/agentsafelabs.com\\\/blog\"],\"url\":\"https:\\\/\\\/agentsafelabs.com\\\/blog\\\/author\\\/waqarjaved\\\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Does Your Agent Framework Choice Actually Matter for Security? We Ran 9,360 Trials to Find Out. - Agentsafelabs","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/agentsafelabs.com\/blog\/does-your-agent-framework-choice-actually-matter-for-security-we-ran-9360-trials-to-find-out\/","og_locale":"en_US","og_type":"article","og_title":"Does Your Agent Framework Choice Actually Matter for Security? We Ran 9,360 Trials to Find Out. - Agentsafelabs","og_description":"A controlled, payload-verified evaluation of seven agentic frameworks says: mostly no \u2014 with one small, honest exception. If you&#8217;re building a tool-using LLM agent today, you&#8217;re choosing among a genuinely crowded field: LangChain, CrewAI, AutoGen, LlamaIndex, the OpenAI Agents SDK, Google&#8217;s Agent Development Kit, Semantic Kernel, or just talking to the model&#8217;s API directly and [&hellip;]","og_url":"https:\/\/agentsafelabs.com\/blog\/does-your-agent-framework-choice-actually-matter-for-security-we-ran-9360-trials-to-find-out\/","og_site_name":"Agentsafelabs","article_published_time":"2026-09-20T18:12:00+00:00","article_modified_time":"2026-09-20T18:18:03+00:00","og_image":[{"width":1672,"height":941,"url":"https:\/\/agentsafelabs.com\/blog\/wp-content\/uploads\/2026\/09\/Does-Your-Agent-Framework-Choice-Actually-Matter-for-Security-We-Ran-9360-Trials-to-Find-Out.png","type":"image\/png"}],"author":"Waqar Javed","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Waqar Javed","Est. reading time":"5 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/agentsafelabs.com\/blog\/does-your-agent-framework-choice-actually-matter-for-security-we-ran-9360-trials-to-find-out\/#article","isPartOf":{"@id":"https:\/\/agentsafelabs.com\/blog\/does-your-agent-framework-choice-actually-matter-for-security-we-ran-9360-trials-to-find-out\/"},"author":{"name":"Waqar Javed","@id":"https:\/\/agentsafelabs.com\/blog\/#\/schema\/person\/76dedaeec309dfebad90b82e70cd80d9"},"headline":"Does Your Agent Framework Choice Actually Matter for Security? We Ran 9,360 Trials to Find Out.","datePublished":"2026-09-20T18:12:00+00:00","dateModified":"2026-09-20T18:18:03+00:00","mainEntityOfPage":{"@id":"https:\/\/agentsafelabs.com\/blog\/does-your-agent-framework-choice-actually-matter-for-security-we-ran-9360-trials-to-find-out\/"},"wordCount":1108,"commentCount":0,"image":{"@id":"https:\/\/agentsafelabs.com\/blog\/does-your-agent-framework-choice-actually-matter-for-security-we-ran-9360-trials-to-find-out\/#primaryimage"},"thumbnailUrl":"https:\/\/agentsafelabs.com\/blog\/wp-content\/uploads\/2026\/09\/Does-Your-Agent-Framework-Choice-Actually-Matter-for-Security-We-Ran-9360-Trials-to-Find-Out.png","keywords":["Agentic System","OWASP Agentic Security","payload-verified evaluation","seven agentic frameworks"],"articleSection":["Blog"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/agentsafelabs.com\/blog\/does-your-agent-framework-choice-actually-matter-for-security-we-ran-9360-trials-to-find-out\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/agentsafelabs.com\/blog\/does-your-agent-framework-choice-actually-matter-for-security-we-ran-9360-trials-to-find-out\/","url":"https:\/\/agentsafelabs.com\/blog\/does-your-agent-framework-choice-actually-matter-for-security-we-ran-9360-trials-to-find-out\/","name":"Does Your Agent Framework Choice Actually Matter for Security? We Ran 9,360 Trials to Find Out. - Agentsafelabs","isPartOf":{"@id":"https:\/\/agentsafelabs.com\/blog\/#website"},"primaryImageOfPage":{"@id":"https:\/\/agentsafelabs.com\/blog\/does-your-agent-framework-choice-actually-matter-for-security-we-ran-9360-trials-to-find-out\/#primaryimage"},"image":{"@id":"https:\/\/agentsafelabs.com\/blog\/does-your-agent-framework-choice-actually-matter-for-security-we-ran-9360-trials-to-find-out\/#primaryimage"},"thumbnailUrl":"https:\/\/agentsafelabs.com\/blog\/wp-content\/uploads\/2026\/09\/Does-Your-Agent-Framework-Choice-Actually-Matter-for-Security-We-Ran-9360-Trials-to-Find-Out.png","datePublished":"2026-09-20T18:12:00+00:00","dateModified":"2026-09-20T18:18:03+00:00","author":{"@id":"https:\/\/agentsafelabs.com\/blog\/#\/schema\/person\/76dedaeec309dfebad90b82e70cd80d9"},"breadcrumb":{"@id":"https:\/\/agentsafelabs.com\/blog\/does-your-agent-framework-choice-actually-matter-for-security-we-ran-9360-trials-to-find-out\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/agentsafelabs.com\/blog\/does-your-agent-framework-choice-actually-matter-for-security-we-ran-9360-trials-to-find-out\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/agentsafelabs.com\/blog\/does-your-agent-framework-choice-actually-matter-for-security-we-ran-9360-trials-to-find-out\/#primaryimage","url":"https:\/\/agentsafelabs.com\/blog\/wp-content\/uploads\/2026\/09\/Does-Your-Agent-Framework-Choice-Actually-Matter-for-Security-We-Ran-9360-Trials-to-Find-Out.png","contentUrl":"https:\/\/agentsafelabs.com\/blog\/wp-content\/uploads\/2026\/09\/Does-Your-Agent-Framework-Choice-Actually-Matter-for-Security-We-Ran-9360-Trials-to-Find-Out.png","width":1672,"height":941,"caption":"Does Your Agent Framework Choice Actually Matter for Security? We Ran 9,360 Trials to Find Out."},{"@type":"BreadcrumbList","@id":"https:\/\/agentsafelabs.com\/blog\/does-your-agent-framework-choice-actually-matter-for-security-we-ran-9360-trials-to-find-out\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/agentsafelabs.com\/blog\/"},{"@type":"ListItem","position":2,"name":"Does Your Agent Framework Choice Actually Matter for Security? We Ran 9,360 Trials to Find Out."}]},{"@type":"WebSite","@id":"https:\/\/agentsafelabs.com\/blog\/#website","url":"https:\/\/agentsafelabs.com\/blog\/","name":"Agentsafelabs","description":"","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/agentsafelabs.com\/blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/agentsafelabs.com\/blog\/#\/schema\/person\/76dedaeec309dfebad90b82e70cd80d9","name":"Waqar Javed","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/secure.gravatar.com\/avatar\/fbceb86c7be592091a2c62a020d43ea3b96a22bf32e19f062870d50e5bbc22b7?s=96&d=mm&r=g","url":"https:\/\/secure.gravatar.com\/avatar\/fbceb86c7be592091a2c62a020d43ea3b96a22bf32e19f062870d50e5bbc22b7?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/fbceb86c7be592091a2c62a020d43ea3b96a22bf32e19f062870d50e5bbc22b7?s=96&d=mm&r=g","caption":"Waqar Javed"},"description":"Waqar Javed is the founder of AgentSafeLabs and a researcher focused on empirical evaluation of AI agent security. safelabs-eval is Apache 2.0 licensed and available on PyPI and GitHub.","sameAs":["https:\/\/agentsafelabs.com\/blog"],"url":"https:\/\/agentsafelabs.com\/blog\/author\/waqarjaved\/"}]}},"_links":{"self":[{"href":"https:\/\/agentsafelabs.com\/blog\/wp-json\/wp\/v2\/posts\/407","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/agentsafelabs.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/agentsafelabs.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/agentsafelabs.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/agentsafelabs.com\/blog\/wp-json\/wp\/v2\/comments?post=407"}],"version-history":[{"count":2,"href":"https:\/\/agentsafelabs.com\/blog\/wp-json\/wp\/v2\/posts\/407\/revisions"}],"predecessor-version":[{"id":409,"href":"https:\/\/agentsafelabs.com\/blog\/wp-json\/wp\/v2\/posts\/407\/revisions\/409"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/agentsafelabs.com\/blog\/wp-json\/wp\/v2\/media\/411"}],"wp:attachment":[{"href":"https:\/\/agentsafelabs.com\/blog\/wp-json\/wp\/v2\/media?parent=407"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/agentsafelabs.com\/blog\/wp-json\/wp\/v2\/categories?post=407"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/agentsafelabs.com\/blog\/wp-json\/wp\/v2\/tags?post=407"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}