Test Improvements

[bigabig]

Now that the api test are merged, we still need to improve some parts, so that testing can be streamlined:

1. Github actions are not working correctly. This is because API tests needs ray, and Github CI machines are not powerful enough to host our ray container (actually, we even disable ray with --disable_ray and RAY_ENABLED=False in the configs for testing). The solution is to host our own ray service on our GPU machines.

2. During development, it should be possible to run the tests locally without any disturbance. Right now, I need to reset my database(s) to have fresh environments for tests. Actually, I want all tests to be run on a seperate sql, elasticsearch, weaviate databases so that my development environment is NOT disturbed. Raffael once prepared a solution for this in run_tests.sh. and a corresponding env.testing.example file. We have to improve this solution, optimally so that everything is configured correctly in our launch.json. Not sure if we really need two new weaviate and ES containers, we just have to make sure that everything is cleaned, after the tests run. In conftest.py, there are already hooks for this (STARTUP_DONE:) and pyteset_sessionfinish().

[bigabig]

this was the old script by raphael:

#!/bin/bash

set -euxo pipefail

# No matter what, before this script stops,
# always remove any docker containers it created to prevent
# them from lying around unused.
function cleanup() {
	docker stop "dats-weaviate-testing-$TEST_CONTAINER_ID" >/dev/null
	docker stop "dats-elasticsearch-testing-$TEST_CONTAINER_ID" >/dev/null
}
trap "cleanup" EXIT

SCRIPT_DIR=$(dirname "${BASH_SOURCE[0]}")

# Read test-specific environment variables from configuration
set -o allexport
source $SCRIPT_DIR/../.env
[[ -f .env.testing ]] && source $SCRIPT_DIR/../.env.testing

# Find the absolute path for the elasticsearch config as that
# is the only thing docker will accept
ELASTICSEARCH_CONFIG=$(realpath -e "$SCRIPT_DIR/../../docker/elasticsearch.yml")

TEST_CONTAINER_ID=$(date +%s)

# Start a temporary elasticsearch database for testing
docker run --detach \
    --name "dats-elasticsearch-testing-$TEST_CONTAINER_ID" \
    --publish "$ELASTICSEARCH_TEST_PORT:9200" \
    --mount type=bind,source="$ELASTICSEARCH_CONFIG",destination=/usr/share/elasticsearch/config/elasticsearch.yml \
    --tmpfs /usr/share/elasticsearch/data \
    --env "xpack.security.enabled=false" \
    --env "discovery.type=single-node" \
    --env "ES_JAVA_OPTS=-Xms512m -Xmx512m" \
    docker.elastic.co/elasticsearch/elasticsearch:8.11.1

# Start a temporary weaviate database for testing
docker run --detach \
    --name "dats-weaviate-testing-$TEST_CONTAINER_ID" \
    --publish "$WEAVIATE_TEST_PORT:8080" \
    --tmpfs /var/lib/weaviate \
    --env "QUERY_DEFAULTS_LIMIT=25" \
    --env "AUTHENTICATION_ANONYMOUS_ACCESS_ENABLED=true" \
    --env "PERSISTENCE_DATA_PATH=/var/lib/weaviate" \
    --env "DEFAULT_VECTORIZER_MODULE=none" \
    --env "CLUSTER_HOSTNAME=node1" \
    --env "LOG_LEVEL=info" \
    semitechnologies/weaviate:1.21.3 \
    --host 0.0.0.0 --port 8080 --scheme http

# Override backend connection ports for the new containers
export WEAVIATE_PORT=$WEAVIATE_TEST_PORT
export ES_PORT=$ELASTICSEARCH_TEST_PORT

# Activate the conda environment for testing
ENV_NAME=dats source _activate_current_env.sh

# Tests will wipe all data; use a temporary repo root
# to protect our development files
REPO_ROOT=$(mktemp -d)
export REPO_ROOT

export PYTHONPATH=.

cd src

pytest "$@"