We are a law firm, based in Mumbai.

Phone: 9004523497 | Email: info@ynzgroup.co.in

7 Legal Hassles to Avoid During AI Implementation

This highlights the caution one must exercise while using and implementing AI tools. AI hallucinations can be real and lead to grave and serious errors. Kerala is the first state to come up with guidelines to use AI in the judicial system through its Policy regarding the usage of Artificial Intelligence Tools in District Judiciary released on 19th July 2025 [6]. While approved AI tools for designated functions are permitted, the policy clearly provides that AI tools shall not be used to arrive at any findings, reliefs, order or judgment under any circumstances as the responsibility for the content and integrity of the judicial order, judgment or any part thereof lies fully with the Judges.

When we use the AI tools, the complete responsibility of using the outcome through such AI tools shall lie with the person using the tool and this can be derived from the concept of Agency under the legal system. Section 182 of the Indian Contract Act, 1872 defines an agent as a person who, in the expressed or implied authority of the principal, acts on behalf of him/her/them. When an AI acts on behalf of a human principal (e.g., an autonomous trading bot), the principal who delegates authority to the AI is likely to remain liable for the AI’s actions within the scope of that authority, as if the principal had acted personally.

Unfortunate incidents of users and teenagers being driven to crimes or suicide have been brought to light on multiple occasions. The most recent being a case wherein the parents of 16-year-old Adam Raine filed a lawsuit in California, alleging that ChatGPT had cultivated an intimate relationship with their son, and provided detailed suicide instructions, and encouraged his death. The lawsuit is filed in a California state court on August 26, 2025.[7]

Suicide is considered a crime under the Indian legal system and in case any tool is considered to be abetting this crime would be held liable under 106 of the BNS. Of course, there could be an argument that the deceased himself or herself voluntarily sought information and thus the tool should be exempt from liability. However, this is unlikely to count as a valid defense especially in cases of minors. Similarly, if a tool provides information on committing a crime, liability could be fastened under the laws.

A German journalist made a shocking discovery earlier this year when he typed his name into Copilot and he was labelled as a criminal who had confessed to crimes[8]. Further, the AI tool described him as a violent criminal and attributed all the crimes that he had reported to him and proceeded to give his name, address and a route map for his home. This is not the only incident and there have been other cases too.

India has passed the Digital Personal Data Protection Act and is on the way of enforcing the legislation. In any case, right to privacy is recognized under Article 21 of the Constitution, and a breach of privacy may attract legal consequences.

We could look at the example of a dataset that was developing using more than 45% images from the US, compared to just 3% from China and India combined[9]. This lack of diversity contributed to many shortcomings of this image recognition algorithm. It interpreted the following :

  • Asian eyes as always blinking
  • Labelled a picture of a traditional US bride dressed in white as bride, dress, woman and wedding but labeled a picture of an Indian bride as performance art and costume and
  • Misidentified the gender/sex of darker-skinned women with a 35% error rate while misidentifying the gender/sex of lighter-skinned men with a 0% error rate.

If a software system is developed with inadequate data then it is possible that such software system could have basic flaws and could have an impact on the warranties made by the programmers and developers while selling the tool. Customers may classify the tool as non-functional completely. Sale of Goods Act, 1930 provides for implied Conditions as to Quality and Fitness. An AI system, whether sold as a software product or a component of hardware, must be reasonably fit for the purpose communicated by the buyer to the seller, and it must be of merchantable quality.

  • If an AI-powered diagnostic tool gives a completely faulty output or has systemic bugs or biases that render its output unreliable, the provider can be held liable for breaches of warranties made by the provider;
  • If a provider describes its AI tool to perform a specific function (e.g., “99% accurate fraud detection”), and the delivered AI fails to match that description, the buyer may have a right to reject the system and claim damages.

The opaque and black box nature of AI systems – i.e. a system which works on a logic which only the system understands- is often criticized.

In one of the earliest cases, seven Houston teachers had successfully challenged the implementation of EVAAS scores which did not properly explain the evaluation method to them. The matter was ultimately settled [11] when the school district agreed to limit usage of the software and also paid $237,000 for attorney’s fees and expenses related to the lawsuit.

This Black Box nature or having an opaque mechanism may be held to be violative of the principles of natural justice in India as well. Transparency of procedure to be followed, is one of the essential principles to be adhered to by all government bodies and any violation will be frowned upon by the Courts [12]. Termination of employees through a score without laying down the criteria would be violative of our labor laws which clearly provide for an enquiry to be conducted before any termination for cause.

7. IP INFRINGEMENT IN GENERATIVE AI:

RESPONSIBLE AI