Post by Founders Law

🚫 The problem with “vibe lawyering” and why AI chatbots can't be your legal department. There’s a lot of noise right now about whether AI can (or should) replace lawyers. So, when this real-life example cropped up recently, we figured it was worth sharing. Someone asked an AI chatbot a seemingly simple question about AI literacy obligations under the EU AI Act: “𝘐𝘴 𝘪𝘵 𝘵𝘳𝘶𝘦 𝘵𝘩𝘢𝘵 𝘰𝘯𝘭𝘺 𝘩𝘪𝘨𝘩-𝘳𝘪𝘴𝘬 𝘈𝘐 𝘶𝘴𝘦 𝘤𝘢𝘴𝘦𝘴 𝘳𝘦𝘲𝘶𝘪𝘳𝘦 𝘈𝘐 𝘭𝘪𝘵𝘦𝘳𝘢𝘤𝘺 𝘵𝘳𝘢𝘪𝘯𝘪𝘯𝘨?” The bot responded confidently:  “𝘠𝘦𝘴 — 𝘶𝘯𝘥𝘦𝘳 𝘵𝘩𝘦 𝘌𝘜 𝘈𝘐 𝘈𝘤𝘵, 𝘵𝘩𝘦 𝘈𝘐 𝘭𝘪𝘵𝘦𝘳𝘢𝘤𝘺 𝘰𝘣𝘭𝘪𝘨𝘢𝘵𝘪𝘰𝘯 𝘢𝘱𝘱𝘭𝘪𝘦𝘴 𝘰𝘯𝘭𝘺 𝘵𝘰 𝘩𝘪𝘨𝘩-𝘳𝘪𝘴𝘬 𝘈𝘐 𝘴𝘺𝘴𝘵𝘦𝘮𝘴.” Sounds convincing, right? It even provided its evidence and reasoning: 𝗖𝗼𝗻𝗳𝗶𝗿𝗺𝗲𝗱: AI Literacy Applies Only to High-Risk Use Cases ✅ 𝗟𝗲𝗴𝗮𝗹 𝗥𝗲𝗳𝗲𝗿𝗲𝗻𝗰𝗲: Article 4(4) of the EU AI Act: “𝘗𝘳𝘰𝘷𝘪𝘥𝘦𝘳𝘴 𝘢𝘯𝘥 𝘥𝘦𝘱𝘭𝘰𝘺𝘦𝘳𝘴 𝘰𝘧 𝘩𝘪𝘨𝘩-𝘳𝘪𝘴𝘬 𝘈𝘐 𝘴𝘺𝘴𝘵𝘦𝘮𝘴 𝘴𝘩𝘢𝘭𝘭 𝘦𝘯𝘴𝘶𝘳𝘦 𝘵𝘩𝘢𝘵 𝘯𝘢𝘵𝘶𝘳𝘢𝘭 𝘱𝘦𝘳𝘴𝘰𝘯𝘴 𝘸𝘩𝘰 𝘢𝘳𝘦 𝘢𝘴𝘴𝘪𝘨𝘯𝘦𝘥 𝘵𝘰 𝘥𝘦𝘷𝘦𝘭𝘰𝘱, 𝘮𝘰𝘯𝘪𝘵𝘰𝘳, 𝘰𝘳 𝘶𝘴𝘦 𝘵𝘩𝘦 𝘈𝘐 𝘴𝘺𝘴𝘵𝘦𝘮𝘴 𝘩𝘢𝘷𝘦 𝘢𝘱𝘱𝘳𝘰𝘱𝘳𝘪𝘢𝘵𝘦 𝘭𝘦𝘷𝘦𝘭𝘴 𝘰𝘧 𝘈𝘐 𝘭𝘪𝘵𝘦𝘳𝘢𝘤𝘺.” 🔍 𝗦𝗰𝗼𝗽𝗲: The obligation is explicitly limited to high-risk AI systems, as defined in Annex III of the Act. 𝗜𝘁 𝗱𝗼𝗲𝘀 𝗻𝗼𝘁 𝗮𝗽𝗽𝗹𝘆 𝘁𝗼: - General-purpose AI (unless used in high-risk systems) - Limited-risk AI systems (e.g., chatbots, recommendation engines) - Minimal-risk AI systems. 𝗧𝗵𝗲 𝗼𝗻𝗹𝘆 𝗽𝗿𝗼𝗯𝗹𝗲𝗺? ❌ 𝗜𝘁’𝘀 𝘄𝗿𝗼𝗻𝗴. Upon checking this against the actual legislation, the chatbot backtracked: “𝘈𝘱𝘰𝘭𝘰𝘨𝘪𝘦𝘴… 𝘐 𝘤𝘢𝘯 𝘤𝘰𝘯𝘧𝘪𝘳𝘮 𝘵𝘩𝘢𝘵 𝘵𝘩𝘦 𝘈𝘐 𝘭𝘪𝘵𝘦𝘳𝘢𝘤𝘺 𝘰𝘣𝘭𝘪𝘨𝘢𝘵𝘪𝘰𝘯 𝘢𝘱𝘱𝘭𝘪𝘦𝘴 𝘵𝘰 𝘢𝘭𝘭 𝘈𝘐 𝘴𝘺𝘴𝘵𝘦𝘮𝘴, 𝘳𝘦𝘨𝘢𝘳𝘥𝘭𝘦𝘴𝘴 𝘰𝘧 𝘳𝘪𝘴𝘬 𝘤𝘭𝘢𝘴𝘴𝘪𝘧𝘪𝘤𝘢𝘵𝘪𝘰𝘯.” And that, ladies and gentlemen... is the risk of “vibe lawyering”: confident answers that sound legal but aren’t 😬 . At Founders Law, we love tech. We’re all for using AI to move faster, scale smarter, and build better, but we also know that legal advice does not run on vibes. Real lawyers?  ✅ Regulated  ✅ Insured  ✅ Can draw on experience ✅ Understand nuance  ✅ Are responsible if they get it wrong So, if you’re building with AI, scaling quickly, or just need legal advice you can actually rely on, we’ve got you covered. Drop us a line: [email protected]