Inference via Whisper. – Dr. Trippers Stuff

lets write a simple script that calls to the whisper large model locally at localhost:5151v1. I will push a button (Left Ctrl) to record and on release it needs to be transcribed and then sent to an ollama endpoint localhost:12434 for the inference wait and receive the answer. Then read that answer to me? Is this the way it is supposed to run or route or flow?
Yes, that flow makes sense. Here is a concise and thorough Python script to achieve this. This script will handle recording audio, sending it to a Whisper model for transcription, then sending the transcribed text to the Ollama endpoint for inference, and finally using text-to-speech (TTS) to read out the response.

We’ll use the following components:
– `sounddevice` for audio recording.
– `requests` for HTTP requests to the Whisper and Ollama endpoints.
– `pyttsx3` for text-to-speech.

### Dependencies

First, ensure you have these dependencies installed:
“`bash
pip install sounddevice requests pyttsx3
“`

### Script

“`python
import sounddevice as sd
import numpy as np
import requests
import pyttsx3
import queue
import keyboard

# Constants
WHISPER_URL = “http://localhost:5151/v1”
OLLAMA_URL = “http://localhost:12434”
SAMPLE_RATE = 16000 # Sample rate for recording
DURATION = 10 # Maximum duration for recording (seconds)

# Initialize text-to-speech engine
tts_engine = pyttsx3.init()

# Queue to hold recorded audio data
audio_queue = queue.Queue()

# Function to record audio
def record_audio():
def callback(indata, frames, time, status):
audio_queue.put(indata.copy())

with sd.InputStream(samplerate=SAMPLE_RATE, channels=1, callback=callback):
print(“Recording… Press Left Ctrl to stop.”)
keyboard.wait(‘left ctrl’)
print(“Stopped recording.”)

# Function to send audio to Whisper for transcription
def transcribe_audio(audio_data):
response = requests.post(
WHISPER_URL,
files={“file”: (“audio.wav”, audio_data, “audio/wav”)}
)
response.raise_for_status()
return response.json().get(‘transcription’, ”)

# Function to send transcription to Ollama for inference
def get_inference(transcription):
response = requests.post(
OLLAMA_URL,
json={“text”: transcription}
)
response.raise_for_status()
return response.json().get(‘response’, ”)

# Function to convert text to speech
def speak(text):
tts_engine.say(text)
tts_engine.runAndWait()

# Main function
def main():
# Record audio
record_audio()

# Retrieve audio data from queue
audio_data = np.concatenate(list(audio_queue.queue))

# Transcribe audio using Whisper
transcription = transcribe_audio(audio_data)

# Get inference from Ollama
response = get_inference(transcription)

# Read the response aloud
speak(response)

if __name__ == “__main__”:
main()
“`

### Explanation

1. **Recording Audio:**
– The `record_audio` function uses `sounddevice` to record audio until the Left Ctrl key is released. The audio data is stored in a queue.

2. **Transcription with Whisper:**
– The `transcribe_audio` function sends the recorded audio data to the Whisper endpoint for transcription and retrieves the transcribed text.

3. **Inference with Ollama:**
– The `get_inference` function sends the transcribed text to the Ollama endpoint and retrieves the inference response.

4. **Text-to-Speech:**
– The `speak` function uses `pyttsx3` to convert the text response to speech and read it aloud.

5. **Main Function:**
– The `main` function orchestrates the entire process by calling the above functions in sequence.

This script assumes that the Whisper and Ollama endpoints are set up correctly and accessible at the specified URLs. Ensure that both services are running and reachable on `localhost`.
no, where do I find a script that works? Are there any sites that have user contributed scripts for llm?
To find user-contributed scripts for large language models (LLMs) and related tasks, you can explore several platforms where developers and researchers share their code. Here are a few notable sites and repositories:

### 1. **GitHub**
GitHub is one of the best places to find and share code for various machine learning and artificial intelligence projects, including LLMs. Here are a few key repositories and organizations to check out:

– **OpenAI**: [OpenAI GitHub](https://github.com/openai)
– **Hugging Face**: [Hugging Face GitHub](https://github.com/huggingface)
– **Google Research**: [Google Research GitHub](https://github.com/google-research)

You can search for specific models or tasks, such as Whisper or Ollama, to find user-contributed scripts.

### 2. **Hugging Face Hub**
Hugging Face provides a vast collection of models, datasets, and scripts shared by the community:

– **Hugging Face Models**: [Hugging Face Models](https://huggingface.co/models)
– **Hugging Face Datasets**: [Hugging Face Datasets](https://huggingface.co/datasets)
– **Hugging Face Spaces**: [Hugging Face Spaces](https://huggingface.co/spaces) (for interactive demos and applications)

### 3. **Kaggle**
Kaggle hosts a variety of machine learning projects and notebooks shared by the community. You can find notebooks demonstrating the use of different models and techniques:

– **Kaggle Notebooks**: [Kaggle Notebooks](https://www.kaggle.com/notebooks)

### 4. **Papers with Code**
Papers with Code links research papers with their corresponding code implementations, which can be very useful for finding state-of-the-art models and their implementations:

– **Papers with Code**: [Papers with Code](https://paperswithcode.com/)

### Example Repositories

Here are a few example repositories that might have what you’re looking for:

– **Whisper Model Implementation**: Search for “whisper model transcription” on GitHub.
– **Ollama Inference Scripts**: Search for “ollama model inference” on GitHub.

For specific implementations, you might need to adapt or combine scripts based on your requirements. If you are still looking for a complete, ready-made script, it’s a good idea to look through these repositories and communities, ask questions, or even contribute your own needs to get help from the community.

If you provide more specific details about what you need, I can help you search for a more targeted script or repository.
what is the standard ollama endpoit port
The standard port for the Ollama endpoint can vary depending on how the server is configured and what defaults are set during installation or deployment. However, it’s common practice to document the specific port used for any service in its official documentation or configuration files.

### Steps to Find the Ollama Endpoint Port

1. **Check Official Documentation**: Review the official Ollama documentation for any mention of default ports.
2. **Configuration Files**: Inspect the configuration files used to set up the Ollama service. This could be in a `config.json`, `.env` file, or similar.
3. **Installation Scripts**: Look at any installation or setup scripts that might specify the port.

### Example

For many services, the default port might be 80 for HTTP or 443 for HTTPS, but custom machine learning inference services often use higher, less common ports (e.g., 5000, 8000, 8080).

If the documentation isn’t readily available, here are some general steps to check the running port:

1. **Check Running Processes**:
“`bash
netstat -tuln | grep LISTEN
“`
This command will show all listening ports on your machine.

2. **Docker Setup**: If Ollama is running inside a Docker container, check the Docker Compose file or run:
“`bash
docker ps
“`
This command lists running containers and their exposed ports.

### Example Configuration Check

If you have an Ollama configuration file (`config.json`, `.env`, etc.), it might look something like this:

“`json
{
“port”: 12434,
“host”: “0.0.0.0”
}
“`

In a `.env` file, it might look like:
“`
PORT=12434
“`

### Conclusion

If you’re unsure about the default port for the Ollama endpoint, and you have no access to the documentation or configuration, you can reach out to the support community or maintainers of the Ollama project for precise information. Without specific details, assuming a port like `12434` (as you mentioned) or `8000`/`8080` might be a good starting point for testing.

C:UsersSarah.condaenvsdeepgramlibsite-packageswhispertranscribe.py:126: UserWarning: FP16 is not supported on CPU; using FP32 instead
warnings.warn(“FP16 is not supported on CPU; using FP32 instead”)
Traceback (most recent call last):
File “C:UsersSarah.condaenvsdeepgramlibsite-packageswhispertranscribe.py”, line 597, in cli
result = transcribe(model, audio_path, temperature=temperature, **args)
File “C:UsersSarah.condaenvsdeepgramlibsite-packageswhispertranscribe.py”, line 133, in transcribe
mel = log_mel_spectrogram(audio, model.dims.n_mels, padding=N_SAMPLES)
File “C:UsersSarah.condaenvsdeepgramlibsite-packageswhisperaudio.py”, line 140, in log_mel_spectrogram
audio = load_audio(audio)
File “C:UsersSarah.condaenvsdeepgramlibsite-packageswhisperaudio.py”, line 58, in load_audio
out = run(cmd, capture_output=True, check=True).stdout
File “C:UsersSarah.condaenvsdeepgramlibsubprocess.py”, line 501, in run
with Popen(*popenargs, **kwargs) as process:
File “C:UsersSarah.condaenvsdeepgramlibsubprocess.py”, line 969, in __init__
self._execute_child(args, executable, preexec_fn, close_fds,
File “C:UsersSarah.condaenvsdeepgramlibsubprocess.py”, line 1438, in _execute_child
hp, ht, pid, tid = _winapi.CreateProcess(executable, args,
FileNotFoundError: [WinError 2] The system cannot find the file specified
Skipping VoiceTTSoutput.wav due to FileNotFoundError: [WinError 2] The system cannot find the file specified

The error you’re encountering suggests that the `whisper` package is attempting to call an external program (likely `ffmpeg`) to process audio, but it’s not able to find the executable. The `FileNotFoundError` indicates that the system cannot locate the required external program.

Here’s how you can address this issue:

### Steps to Resolve the Error

1. **Install FFmpeg**: `whisper` requires `ffmpeg` to process audio files. Download and install FFmpeg from [ffmpeg.org](https://ffmpeg.org/download.html). Make sure to add FFmpeg to your system’s PATH.

2. **Verify FFmpeg Installation**:
– After installing FFmpeg, open a new terminal or command prompt and type `ffmpeg -version` to ensure it’s correctly installed and accessible.

3. **Add FFmpeg to PATH**:
– If FFmpeg is not found, you might need to add it to your system’s PATH. Here’s how to do it:
– **Windows**:
1. Download FFmpeg and extract the contents.
2. Copy the path to the `bin` directory (e.g., `C:pathtoffmpegbin`).
3. Open the Start Menu, search for “Environment Variables” and select “Edit the system environment variables”.
4. In the System Properties window, click “Environment Variables”.
5. In the Environment Variables window, find the `Path` variable in the “System variables” section, select it, and click “Edit”.
6. Click “New” and paste the path to the `bin` directory of FFmpeg.
7. Click “OK” to close all windows.

### Example Script Adjustments

After ensuring FFmpeg is installed and correctly configured, your script should be able to run without the `FileNotFoundError`.

### Script Verification

Let’s update and verify your script to ensure it works:

1. **Check FFmpeg Installation**:

Open a command prompt and run:
“`sh
ffmpeg -version
“`

This should display the version of FFmpeg installed.

2. **Sample Python Script**:

If you have a Python script that uses Whisper, ensure it’s correctly set up to handle audio files and invoke FFmpeg. Here’s a simplified example to demonstrate this:

“`python
import subprocess
import whisper

def transcribe_audio(audio_path):
model = whisper.load_model(“base”)

# Check if FFmpeg is installed
try:
subprocess.run([“ffmpeg”, “-version”], check=True, capture_output=True)
except FileNotFoundError:
raise RuntimeError(“FFmpeg is not installed or not found in PATH”)

result = model.transcribe(audio_path)
return result[‘text’]

if __name__ == “__main__”:
audio_path = “path/to/your/audio/file.wav” # Specify the path to your audio file
transcription = transcribe_audio(audio_path)
print(“Transcription:”, transcription)
“`

### Conclusion

By installing FFmpeg and ensuring it’s accessible via your system’s PATH, you should be able to resolve the `FileNotFoundError` and successfully run your Whisper transcription script. If you encounter further issues, make sure to verify all paths and dependencies are correctly set up.
ffmpeg-pythonffmpeg-python
To resolve the issue using `ffmpeg-python`, you can use this library to handle audio processing directly within your script, eliminating the need for external FFmpeg commands.

First, install the `ffmpeg-python` package:
“`bash
pip install ffmpeg-python
“`

Here’s how you can integrate `ffmpeg-python` with the Whisper model for transcription:

### Sample Script Using `ffmpeg-python`

“`python
import ffmpeg
import numpy as np
import whisper
import sounddevice as sd
import pyttsx3
import queue
import keyboard

# Constants
WHISPER_URL = “http://localhost:5151/v1”
OLLAMA_URL = “http://localhost:12434”
SAMPLE_RATE = 16000 # Sample rate for recording
CHANNELS = 1 # Number of audio channels
DURATION = 10 # Maximum duration for recording (seconds)

# Initialize text-to-speech engine
tts_engine = pyttsx3.init()

# Queue to hold recorded audio data
audio_queue = queue.Queue()

# Function to record audio
def record_audio():
def callback(indata, frames, time, status):
audio_queue.put(indata.copy())

with sd.InputStream(samplerate=SAMPLE_RATE, channels=CHANNELS, callback=callback):
print(“Recording… Press Left Ctrl to stop.”)
keyboard.wait(‘left ctrl’)
print(“Stopped recording.”)

# Function to convert recorded audio to a format suitable for Whisper
def convert_audio_to_wav(audio_data):
audio_path = ‘temp_audio.wav’
wav_data = np.concatenate(list(audio_queue.queue))
ffmpeg.input(‘pipe:0′, format=’f32le’, ac=CHANNELS, ar=SAMPLE_RATE).output(audio_path).run(input=wav_data.tobytes(), overwrite_output=True)
return audio_path

# Function to transcribe audio using Whisper
def transcribe_audio(audio_path):
model = whisper.load_model(“base”)
result = model.transcribe(audio_path)
return result[‘text’]

# Function to convert text to speech
def speak(text):
tts_engine.say(text)
tts_engine.runAndWait()

# Main function
def main():
# Record audio
record_audio()

# Convert recorded audio to WAV format
audio_path = convert_audio_to_wav(audio_queue.queue)

# Transcribe audio using Whisper
transcription = transcribe_audio(audio_path)

# Get inference from Ollama
response = get_inference(transcription)

# Read the response aloud
speak(response)

if __name__ == “__main__”:
main()
“`

### Explanation

1. **Audio Recording**: Records audio using `sounddevice`.
2. **Convert to WAV**: Uses `ffmpeg-python` to convert the recorded audio data to WAV format.
3. **Whisper Transcription**: Uses the Whisper model to transcribe the audio.
4. **Inference with Ollama**: Sends the transcription to the Ollama endpoint for inference.
5. **Text-to-Speech**: Uses `pyttsx3` to read the response aloud.

This script should help you avoid `FileNotFoundError` by directly integrating `ffmpeg-python` for audio processing.
Using `ffmpeg-python` can help you avoid directly interacting with the `ffmpeg` command-line tool, making it easier to work with FFmpeg in Python. `ffmpeg-python` is a Python library that provides a fluent interface for creating FFmpeg command lines.

Here’s how you can integrate `ffmpeg-python` into your script to handle the audio processing needed by Whisper:

### Installation

First, install the `ffmpeg-python` library:
“`bash
pip install ffmpeg-python
“`

### Example Script Using `ffmpeg-python`

Here’s an updated version of your script that incorporates `ffmpeg-python` to handle audio processing:

“`python
import ffmpeg
import whisper
import numpy as np
import sounddevice as sd
import queue
import keyboard
import requests
import pyttsx3

# Constants
WHISPER_URL = “http://localhost:5151/v1”
OLLAMA_URL = “http://localhost:12434”
SAMPLE_RATE = 16000 # Sample rate for recording
CHANNELS = 1
DURATION = 10 # Maximum duration for recording (seconds)

# Initialize text-to-speech engine
tts_engine = pyttsx3.init()

# Queue to hold recorded audio data
audio_queue = queue.Queue()

# Function to record audio
def record_audio():
def callback(indata, frames, time, status):
audio_queue.put(indata.copy())

with sd.InputStream(samplerate=SAMPLE_RATE, channels=CHANNELS, callback=callback):
print(“Recording… Press Left Ctrl to stop.”)
keyboard.wait(‘left ctrl’)
print(“Stopped recording.”)

# Function to save audio from the queue to a file
def save_audio_file(filename):
audio_data = np.concatenate(list(audio_queue.queue))
ffmpeg.input(‘pipe:0′, format=’f32le’, ac=CHANNELS, ar=SAMPLE_RATE).output(filename).run(input=audio_data.tobytes())

# Function to send audio to Whisper for transcription
def transcribe_audio(audio_path):
response = requests.post(
WHISPER_URL,
files={“file”: (audio_path, open(audio_path, “rb”), “audio/wav”)}
)
response.raise_for_status()
return response.json().get(‘transcription’, ”)

# Function to convert text to speech
def speak(text):
tts_engine.say(text)
tts_engine.runAndWait()

# Main function
def main():
# Record audio
record_audio()

# Save recorded audio to file
audio_file = “recorded_audio.wav”
save_audio_file(audio_file)

# Transcribe audio using Whisper
transcription = transcribe_audio(audio_file)
print(“Transcription:”, transcription)

# Get inference from Ollama
response = get_inference(transcription)
print(“Response:”, response)

# Read the response aloud
speak(response)

if __name__ == “__main__”:
main()
“`

### Explanation

1. **Recording Audio**:
– The `record_audio` function captures audio until the Left Ctrl key is released. The recorded audio is stored in a queue.

2. **Saving Audio with `ffmpeg-python`**:
– The `save_audio_file` function uses `ffmpeg-python` to process and save the audio data from the queue to a file. It converts the audio data to the appropriate format expected by Whisper.

3. **Transcription with Whisper**:
– The `transcribe_audio` function sends the recorded audio file to the Whisper endpoint for transcription and retrieves the transcribed text.

4. **Inference with Ollama**:
– The `get_inference` function sends the transcribed text to the Ollama endpoint and retrieves the response.

5. **Text-to-Speech**:
– The `speak` function uses `pyttsx3` to convert the text response to speech and read it aloud.

### Conclusion

By integrating `ffmpeg-python`, you can handle audio processing directly within Python, streamlining the process and avoiding potential issues with external command-line calls. Ensure that all services (Whisper and Ollama) are running and accessible at the specified endpoints.
python app.py
Recording… Press Left Ctrl to stop.
Stopped recording.
Traceback (most recent call last):
File “D:EveBusinessVoiceTTSapp.py”, line 77, in
main()
File “D:EveBusinessVoiceTTSapp.py”, line 65, in main
audio_path = convert_audio_to_wav(audio_queue.queue)
File “D:EveBusinessVoiceTTSapp.py”, line 36, in convert_audio_to_wav
ffmpeg.input(‘pipe:0′, format=’f32le’, ac=CHANNELS, ar=SAMPLE_RATE).output(audio_path).run(input=wav_data.tobytes(), overwrite_output=True)
File “C:UsersSarah.condaenvsdeepgramlibsite-packagesffmpeg_run.py”, line 313, in run
process = run_async(
File “C:UsersSarah.condaenvsdeepgramlibsite-packagesffmpeg_run.py”, line 284, in run_async
return subprocess.Popen(
File “C:UsersSarah.condaenvsdeepgramlibsubprocess.py”, line 969, in __init__
self._execute_child(args, executable, preexec_fn, close_fds,
File “C:UsersSarah.condaenvsdeepgramlibsubprocess.py”, line 1438, in _execute_child
hp, ht, pid, tid = _winapi.CreateProcess(executable, args,
FileNotFoundError: [WinError 2] The system cannot find the file specified
The error you’re encountering (`FileNotFoundError: [WinError 2] The system cannot find the file specified`) indicates that the script cannot find the FFmpeg executable. This often happens when FFmpeg is not installed correctly or is not in your system’s PATH.

### Steps to Resolve

1. **Download and Install FFmpeg**:
– Download the FFmpeg zip file for your operating system from [ffmpeg.org](https://ffmpeg.org/download.html).
– Extract the contents of the zip file to a directory (e.g., `C:ffmpeg`).

2. **Add FFmpeg to System PATH**:
– Copy the path to the `bin` directory inside the extracted folder (e.g., `C:ffmpegbin`).
– Add this path to your system’s PATH environment variable:
– **Windows**:
1. Open the Start Menu and search for “Environment Variables”, then select “Edit the system environment variables”.
2. In the System Properties window, click on the “Environment Variables” button.
3. In the Environment Variables window, find the `Path` variable in the “System variables” section, select it, and click “Edit”.
4. Click “New” and paste the path to the `bin` directory of FFmpeg.
5. Click “OK” to close all windows.

3. **Verify FFmpeg Installation**:
– Open a new command prompt and type `ffmpeg -version` to verify that FFmpeg is installed correctly and accessible from the command line.

### Updated Script

Let’s modify the script slightly to ensure that FFmpeg is invoked correctly using `ffmpeg-python`.

“`python
import ffmpeg
import numpy as np
import whisper
import sounddevice as sd
import pyttsx3
import queue
import keyboard

# Initialize text-to-speech engine
tts_engine = pyttsx3.init()

# Queue to hold recorded audio data
audio_queue = queue.Queue()

# Function to record audio
def record_audio():
def callback(indata, frames, time, status):
audio_queue.put(indata.copy())

with sd.InputStream(samplerate=SAMPLE_RATE, channels=CHANNELS, callback=callback):
print(“Recording… Press Left Ctrl to stop.”)
keyboard.wait(‘left ctrl’)
print(“Stopped recording.”)

# Function to convert recorded audio to a format suitable for Whisper
def convert_audio_to_wav(audio_queue):
audio_path = ‘temp_audio.wav’
wav_data = np.concatenate(list(audio_queue.queue))
# Convert to int16
wav_data = (wav_data * np.iinfo(np.int16).max).astype(np.int16)
process = (
ffmpeg
.input(‘pipe:0′, format=’s16le’, ac=CHANNELS, ar=SAMPLE_RATE)
.output(audio_path, format=’wav’)
.overwrite_output()
.run_async(pipe_stdin=True)
)
process.stdin.write(wav_data.tobytes())
process.stdin.close()
process.wait()
return audio_path

# Function to transcribe audio using Whisper
def transcribe_audio(audio_path):
model = whisper.load_model(“base”)
result = model.transcribe(audio_path)
return result[‘text’]

# Function to convert text to speech
def speak(text):
tts_engine.say(text)
tts_engine.runAndWait()

# Main function
def main():
# Record audio
record_audio()

# Convert recorded audio to WAV format
audio_path = convert_audio_to_wav(audio_queue)

# Transcribe audio using Whisper
transcription = transcribe_audio(audio_path)

# Get inference from Ollama
response = get_inference(transcription)

# Read the response aloud
speak(response)

if __name__ == “__main__”:
main()
“`

### Explanation

1. **FFmpeg Installation and PATH**: Ensure FFmpeg is installed and added to the system PATH.
2. **FFmpeg-Python Usage**: Use `ffmpeg-python` to handle the audio conversion, ensuring the correct format and execution.
3. **Audio Data Handling**: Convert the audio data to the `int16` format before passing it to FFmpeg, as it expects PCM data in this format.

By following these steps and using the updated script, you should be able to resolve the `FileNotFoundError` and successfully process and transcribe your audio.