Whisper-Hindi2Hinglish-Apex

Whisper-Hindi2Hinglish-Apex:#

• GITHUB LINK: github link
• SPEECH-TO-TEXT ARENA: Speech-To-Text Arena

Table of Contents:#

• Key Features
• Training
  • Data
  • Finetuning
• Usage
• Performance Overview
  • Qualitative Performance Overview
  • Quantitative Performance Overview
• Miscellaneous

Key Features:#

1. Hinglish as a language: Native ability to transcribe hindi into hinglish, making it easier to understand and process.
2. Faster Inference: The model achieves SOTA performance while being upto 8x faster.
3. Performance Increase: ~42% average performance increase versus pretrained model across benchmarking datasets.
4. Ranked #1 on Speech-To-Text Arena, beating competitors on transcription preference metric.

Training:#

Data:#

• Duration: A total of ~700 Hrs of noisy Indian-accented Hindi audios.
• Collection: Curated using a collection of open-source and proprietary dataset.
• Labelling: Automated labelling using SOTA model, with manual human correction.

Finetuning:#

• Novel Finetuning Techniques: Advanced fine-tuning techniques were used to specifically target and increase performance on noisy Indian-accented audios.
  • Custom Dynamic Layer Freezing: Identifying most active layers during inference and perfomed targeted training of those layers.
  • Dataset Augmentations: Devised data augmentation techniques to increase model robustness.

Performance Overview#

Qualitative Performance Overview#

Quantitative Performance Overview#

Note:

• To accurately measure Hinglish transcription performance, the original Hindi ground truth for each dataset was first transliterated to Hinglish. The WER scores below were calculated against this transliterated reference text.
• To check our model's real-world performance against other SOTA models please head to our Speech-To-Text Arena arena space.

Usage:#

Using Transformers#

• To run the model, first install the Transformers library

- The model can be used with the [`pipeline`](https://huggingface.co/docs/transformers/main_classes/pipelines#transformers.AutomaticSpeechRecognitionPipeline)
class to transcribe audios of arbitrary length:

import torch from transformers import AutoModelForSpeechSeq2Seq, AutoProcessor, pipeline from datasets import load_dataset

Set device (GPU if available, otherwise CPU) and precision#

device = "cuda:0" if torch.cuda.is_available() else "cpu" torch_dtype = torch.float16 if torch.cuda.is_available() else torch.float32

Specify the pre-trained model ID#

model_id = "Oriserve/Whisper-Hindi2Hinglish-Apex"

Load the speech-to-text model with specified configurations#

model = AutoModelForSpeechSeq2Seq.from_pretrained( model_id, torch_dtype=torch_dtype, # Use appropriate precision (float16 for GPU, float32 for CPU) low_cpu_mem_usage=True, # Optimize memory usage during loading use_safetensors=True # Use safetensors format for better security ) model.to(device) # Move model to specified device

Load the processor for audio preprocessing and tokenization#

processor = AutoProcessor.from_pretrained(model_id)

Create speech recognition pipeline#

pipe = pipeline( "automatic-speech-recognition", model=model, tokenizer=processor.tokenizer, feature_extractor=processor.feature_extractor, torch_dtype=torch_dtype, device=device, generate_kwargs={ "task": "transcribe", # Set task to transcription "language": "en" # Specify English language } )

Process audio file and print transcription#

sample = "sample.wav" # Input audio file path result = pipe(sample) # Run inference print(result["text"]) # Print transcribed text

#### Using Flash Attention 2

Flash-Attention 2 can be used to make the transcription fast. If your GPU supports Flash-Attention you can use it by, first installing Flash Attention:

• Once installed you can then load the model using the below code:

model = AutoModelForSpeechSeq2Seq.from_pretrained(model_id, torch_dtype=torch_dtype, low_cpu_mem_usage=True, attn_implementation="flash_attention_2")

Using the OpenAI Whisper module#

• First, install the openai-whisper library

- Convert the huggingface checkpoint to a pytorch model

import torch from transformers import AutoModelForSpeechSeq2Seq import re from tqdm import tqdm from collections import OrderedDict import json

Load parameter name mapping from HF to OpenAI format#

with open('convert_hf2openai.json', 'r') as f: reverse_translation = json.load(f)

reverse_translation = OrderedDict(reverse_translation)

def save_model(model, save_path): def reverse_translate(current_param): # Convert parameter names using regex patterns for pattern, repl in reverse_translation.items(): if re.match(pattern, current_param): return re.sub(pattern, repl, current_param)

# Extract model dimensions from config config = model.config model_dims = { "n_mels": config.num_mel_bins, # Number of mel spectrogram bins "n_vocab": config.vocab_size, # Vocabulary size "n_audio_ctx": config.max_source_positions, # Max audio context length "n_audio_state": config.d_model, # Audio encoder state dimension "n_audio_head": config.encoder_attention_heads, # Audio encoder attention heads "n_audio_layer": config.encoder_layers, # Number of audio encoder layers "n_text_ctx": config.max_target_positions, # Max text context length "n_text_state": config.d_model, # Text decoder state dimension "n_text_head": config.decoder_attention_heads, # Text decoder attention heads "n_text_layer": config.decoder_layers, # Number of text decoder layers }

# Convert model state dict to Whisper format original_model_state_dict = model.state_dict() new_state_dict = {}

for key, value in tqdm(original_model_state_dict.items()): key = key.replace("model.", "") # Remove 'model.' prefix new_key = reverse_translate(key) # Convert parameter names if new_key is not None: new_state_dict[new_key] = value

# Create final model dictionary pytorch_model = {"dims": model_dims, "model_state_dict": new_state_dict}

# Save converted model torch.save(pytorch_model, save_path)

Load Hugging Face model#

model_id = "Oriserve/Whisper-Hindi2Hinglish-Apex" model = AutoModelForSpeechSeq2Seq.from_pretrained( model_id, low_cpu_mem_usage=True, # Optimize memory usage use_safetensors=True # Use safetensors format )

Convert and save model#

model_save_path = "Whisper-Hindi2Hinglish-Apex.pt" save_model(model,model_save_path)

- Transcribe

import whisper

Load converted model with Whisper and transcribe#

model = whisper.load_model("Whisper-Hindi2Hinglish-Apex.pt") result = model.transcribe("sample.wav") print(result["text"])

### Miscellaneous
This model is from a family of transformers-based ASR models trained by Oriserve. To compare this model against other models from the same family or other SOTA models please head to our [Speech-To-Text Arena](https://huggingface.co/spaces/Oriserve/ASR_arena). To learn more about our other models, and other queries regarding AI voice agents you can reach out to us at our email [ai-team@oriserve.com](ai-team@oriserve.com)