The integration[1] of Artificial Intelligence (AI) into India’s commercial and public life is accelerating at an unprecedented pace, and naturally so, as AI tools have an unmatched speed and seem to be perfect, when we get precise project reports, flawless poems and beautiful presentations.
AI has also contributed significantly to the revenue and tech companies that have called out AI revenue specifically in their Q2 results.[2] As acknowledged by our Ministry of Electronics and Information Technology, Artificial intelligence (A.I.) is no longer the realm of science fiction but a practical software tool used to help millions of people every day. India has been reported to be among the top 3 locations in the globe (after the US and China) for the development and iteration of AI related technologies and has been praised as having over 58% of its technological uses of AI in the implementation stage, i.e beyond pilot and test projects.[3]
However, not all is perfect and there are possibilities of liabilities which may revolve due to usage and implementation of AI. While the government is yet to enact a dedicated law and comprehensive guidelines, this article examines the liabilities that could arise from algorithmic failures under the existing legal framework of India.
The Algorithmic Blame Game
1. Discrimination through an AI system:
Imagine you have a credit company which needs to look into huge piles of data and allot a score to the loan applications made by various individuals. You may feel that using an AI tool to get a credit rating (which will supplement the actual examination by a human being) could save time. However, if the criteria of setting the scores are not transparent and properly set, the system could come out with arbitrary and discriminatory scores, which could lead to discrimination issues especially in a diverse country like India.
Recently, it was widely reported that SafeRent Solutions, an AI-driven tenant screening tool, settled a USD 2.3 million class-action lawsuit after allegations of discriminatory practices. It was alleged by the applicant – Mary Louis that she was unable to get a house on rent due to consistent poor scores being awarded to her. She was one of a class of more than 400 Black and Hispanic tenants in Massachusetts who use housing vouchers and said their rental applications were rejected because of their SafeRent score. In 2022, they came together to sue the company and the lawsuit alleged that SafeRent’s algorithm disproportionately scored low-income applicants and people of colour, particularly those using housing vouchers. These scores led to unjust denials of housing applications, a clear violation of the Fair Housing Act.[4] The matter was amicably settled in 2024 by the parties out of the court as per the monetary settlement.
Indian constitution guarantees equality before law to all its citizens under Article 14 and Article 16 as well. Thus, cases of system driven discrimination could very well have serious legal impact, especially if the discrimination is through a public institution.
2. AI Hallucination:
It seems that AI tools have a compelling need to serve and come up with answers when some information is sought repeatedly. If a tool is questioned repeatedly by a user, the tool may, get hallucinated and generate information that is false and fabricated. Recently, an incident came to light, when the Bengaluru bench of the Income Tax Appellate Tribunal (“ITAT”) passed an interesting order in Buckeye Trust v. Principal Commissioner of Income Tax (ITA No.1051/Bang/2024). It ascertained the taxability of a transaction on the basis of judgements which simply did not exist! The Tribunal relied on four judgements – of the Supreme Court and the Madras High Court – of which two consisted of completely fabricated case names and citations, one had a fabricated case name accompanied by the citation of a different case, and another led to a case for an irrelevant proposition of law.[5]
This highlights the caution one must exercise while using and implementing AI tools. AI hallucinations can be real and lead to grave and serious errors. Kerala is the first state to come up with guidelines to use AI in the judicial system through its Policy regarding the usage of Artificial Intelligence Tools in District Judiciary released on 19th July 2025 [6]. While approved AI tools for designated functions are permitted, the policy clearly provides that AI tools shall not be used to arrive at any findings, reliefs, order or judgment under any circumstances as the responsibility for the content and integrity of the judicial order, judgment or any part thereof lies fully with the Judges.
When we use the AI tools, the complete responsibility of using the outcome through such AI tools shall lie with the person using the tool and this can be derived from the concept of Agency under the legal system. Section 182 of the Indian Contract Act, 1872 defines an agent as a person who, in the expressed or implied authority of the principal, acts on behalf of him/her/them. When an AI acts on behalf of a human principal (e.g., an autonomous trading bot), the principal who delegates authority to the AI is likely to remain liable for the AI’s actions within the scope of that authority, as if the principal had acted personally.
3. Lack of Empathy:
AI systems like ChatGPT or Claude are trained to provide answers and designed to be helpful, polite, and user aligned. However, this alignment can sometimes go too far leading the AI to mirror the user’s opinions or assumptions instead of offering an accurate, empathetic or balanced view.
Unfortunate incidents of users and teenagers being driven to crimes or suicide have been brought to light on multiple occasions. The most recent being a case wherein the parents of 16-year-old Adam Raine filed a lawsuit in California, alleging that ChatGPT had cultivated an intimate relationship with their son, and provided detailed suicide instructions, and encouraged his death. The lawsuit is filed in a California state court on August 26, 2025.[7]
Suicide is considered a crime under the Indian legal system and in case any tool is considered to be abetting this crime would be held liable under 106 of the BNS. Of course, there could be an argument that the deceased himself or herself voluntarily sought information and thus the tool should be exempt from liability. However, this is unlikely to count as a valid defense especially in cases of minors. Similarly, if a tool provides information on committing a crime, liability could be fastened under the laws.
4. Data Privacy issues:
While using and analyzing data, an AI tool may unknowingly cross the lines and disclose personal information to the users. Right to privacy is a recognized right across the world, and anyone misusing any person’s personal information may be sued for breach of privacy under data protection laws.
A German journalist made a shocking discovery earlier this year when he typed his name into Copilot and he was labelled as a criminal who had confessed to crimes[8]. Further, the AI tool described him as a violent criminal and attributed all the crimes that he had reported to him and proceeded to give his name, address and a route map for his home. This is not the only incident and there have been other cases too.
India has passed the Digital Personal Data Protection Act and is on the way of enforcing the legislation. In any case, right to privacy is recognized under Article 21 of the Constitution, and a breach of privacy may attract legal consequences.
5. Bias:
We live in a diverse world of varied languages, races, skin tones, features and colors. Facial recognition tools have often faced various issues and have brought up a chilling effect. When a tool is trained without the proper diverse data, the impact can be strange and often leave the users with a complex feeling, and sometime a chilling effect.
We could look at the example of a dataset that was developing using more than 45% images from the US, compared to just 3% from China and India combined[9]. This lack of diversity contributed to many shortcomings of this image recognition algorithm. It interpreted the following :
- Asian eyes as always blinking
- Labelled a picture of a traditional US bride dressed in white as bride, dress, woman and wedding but labeled a picture of an Indian bride as performance art and costume and
- Misidentified the gender/sex of darker-skinned women with a 35% error rate while misidentifying the gender/sex of lighter-skinned men with a 0% error rate.
If a software system is developed with inadequate data then it is possible that such software system could have basic flaws and could have an impact on the warranties made by the programmers and developers while selling the tool. Customers may classify the tool as non-functional completely. Sale of Goods Act, 1930 provides for implied Conditions as to Quality and Fitness. An AI system, whether sold as a software product or a component of hardware, must be reasonably fit for the purpose communicated by the buyer to the seller, and it must be of merchantable quality.
- If an AI-powered diagnostic tool gives a completely faulty output or has systemic bugs or biases that render its output unreliable, the provider can be held liable for breaches of warranties made by the provider;
- If a provider describes its AI tool to perform a specific function (e.g., “99% accurate fraud detection”), and the delivered AI fails to match that description, the buyer may have a right to reject the system and claim damages.
6. The ‘black box’ nature of Algorithmic Decision-Making and XAI:
Explainable AI is defined by UNESCO as systems, algorithms, and models with the ability to explain their rationale for decisions, characterize the strengths and weaknesses of their decision-making process, and convey an understanding of how they will behave in the future, and is the buzz word today [10].
The opaque and black box nature of AI systems – i.e. a system which works on a logic which only the system understands- is often criticized.
In one of the earliest cases, seven Houston teachers had successfully challenged the implementation of EVAAS scores which did not properly explain the evaluation method to them. The matter was ultimately settled [11] when the school district agreed to limit usage of the software and also paid $237,000 for attorney’s fees and expenses related to the lawsuit.
This Black Box nature or having an opaque mechanism may be held to be violative of the principles of natural justice in India as well. Transparency of procedure to be followed, is one of the essential principles to be adhered to by all government bodies and any violation will be frowned upon by the Courts [12]. Termination of employees through a score without laying down the criteria would be violative of our labor laws which clearly provide for an enquiry to be conducted before any termination for cause.

7. IP INFRINGEMENT IN GENERATIVE AI:
This is another big hassle, and legal challenges have been initiated in courts particularly before the Delhi High Court. ANI Media Pvt. Ltd. has sued OpenAI Inc and the Court has also issued summons in this matter. There are instances where generative AI models were trained on copyrighted material and the AI tool subsequently reproduced that content, raising direct issues under the Copyright Act, 1957. One must be careful while using materials, images from AI tools directly as the same may lead to huge intellectual property disputes.
RESPONSIBLE AI
As organizations embrace AI to enhance efficiency and decision-making, it is crucial to remember that technology is only as responsible as the humans who design, train, and deploy it. The challenges ranging from algorithmic bias and hallucinations to data privacy lapses and copyright concern reflects the need for a cautious, ethical, and legally compliant approach. In essence, the goal should not be to replace human judgment, but to collaborate it with system that reflect our high standards of integrity, legality and respect for law.

- The article reflects the general work of the author on the date of publication and the views expressed are personal. No reader should act on any statement contained herein without seeking detailed professional advice
- HCLTech reports fastest Q2 growth in five years, calls AI revenue in a first
- 11ab.pdf
- She didn’t get an apartment because of an AI-generated score – and sued to help others avoid the same fate | Artificial intelligence (AI) | The Guardian
- Phantom Precedents: When AI Hallucinations Invade Judicial Reasoning – SpicyIP
- Kerala_HC_AI_Guidelines.pdf
- Legal team that sued OpenAI after teen’s death calls on company to do more on safety – The Hindu
- committees.parliament.uk/publications/46800/documents/240861/default/
- UNESCO: Global toolkit on AI and the rule of law for the judiciary : Global toolkit on AI and the rule of law for the judiciary – UNESCO Digital Library
- Ibid 9
- Federal Suit Settlement: End of Value-Added Measures for Teacher Termination in Houston | American Federation of Teacher
- More information on principles of natural justice can be found here: Microsoft Word – 15-Third article- Audi Alteram partem – 12th October_Final.docx
For any feedback or response on this article, the authors can be reached on aarti.banerjee@ynzgroup.co.in and kedar.gurjar@ynzgroup.co.in

Author : Aarti Banerjee
Aarti is a Partner – Corporate Legal Advisory.
Aarti is experienced in corporate legal matters having specialization in drafting, vetting and negotiation of agreements. By qualification she is an advocate and a solicitor.

Co-Author : Kedar Gurjar
Kedar is an Associate at YNZ Legal.
By qualification he is Bachelor of Commerce and Bachelor of Law from G.J. Advani Law college, Bandra, Mumbai University.