Make it possible for private AI models to get tested.

[reisepass]

There are many AI companies with significant revenue where the AI model being served can not be openly purchased by the public but must first go through a B2B sales and KYB process.

Allowing this would require

• Validators to KYB with the private AI model providers
• An encrypted or permissioned storage of the prompt and scoring audit logs
• Validators being distributed private API credentials

[Isilvorn]

Some additional considerations for evaluating private models that we might want to consider are as follows. Some of these will probably be applicable generally, also.

Enabling the evaluation of private AI models within a decentralized scoring system introduces a few additional challenges that we might want to address:

Confidentiality & IP Protection
• Validators may need to KYB and sign NDAs with model providers.
• Access should be controlled via token-based permissions or time-limited API credentials.

Validation Protocol & Trust
• Model authenticity should be verifiable (e.g., digital signatures).
• Zero-knowledge proofs (ZK-proofs) could enable validation without exposing proprietary details.

Data Privacy & Compliance
• Logs should be encrypted or permissioned to prevent exposure of proprietary prompts.
• Compliance with GDPR/CCPA may be required if real user data is involved.

Auditability for Dispute Resolution (plus it would probably be a generally good practice)
• All evaluations should be immutably recorded on-chain for transparency.
• A dispute resolution process is needed to handle disagreements over evaluations.

Technical Integration & Infrastructure
• Private API credentials must be securely distributed and rotated.
• Model providers may run evaluations on their own infrastructure, requiring verifiable audit logs.

Scalability & Cost
• Running evaluations on large models could be expensive if it’s not paid for by the private model creators—validators may need incentives.
• Trade-offs between real-time and batch evaluations should be considered.

Benchmark Evolution & Fairness
• Benchmarks should be dynamic to prevent overfitting and remain representative of real-world tasks.
• Standardized evaluation protocols can improve cross-model comparability.

Community & Governance
• A permissioned validator system may be necessary for private models.
• Governance should prevent any single entity from manipulating results.

These are my initial thoughts. If anyone else has additional ideas, please add.

[saifmir700]

Here is an idea that I think takes both Ruben and Jason's concerns into consideration. Please let me know your thoughts:
Possible solution: Secure Storage of Logs Using Encrypted Permissioned Databases

Encrypted storage prevents exposure of sensitive prompts & responses.
Can be implemented using zero-knowledge proofs or secure multiparty computation (though there is increased computational overhead) to allow auditing without revealing proprietary details.
This would also be compliant with the EU's GDPR.

Accounts for data privacy, compliance, auditability, and IP protection. I have a few other ideas, such as distributing private API credentials to validators, but please provide feedback on this first. Thank you!