Back to limesurvey.org

Welcome to the LimeSurvey Community Forum

Ask the community, share ideas, and connect with other LimeSurvey users!

Performance concerns: Custom JS searchable dropdown with external resource file

More
2 weeks 6 days ago #273720 by Louuuu
Performance concerns: Custom JS searchable dropdown with external resource file was created by Louuuu
Please help us help you and fill where relevant: 
Your LimeSurvey version: 6.15.17
Own server or LimeSurvey hosting: LimeSurvey hosting
Survey theme/template: fruity_twenty_three
==================
Hello,I am currently working on LimeSurvey version 6.15.17, hosted on a university server.I have designed a survey where several questions use custom JavaScript to create searchable dropdown menus with autocompletion. These scripts pull data from a resource file uploaded to the survey.A key point is that the list contains more than 5,000 entries, which makes the file quite significant for a client-side script.Here is the code I am using for one of these questions:// =========================================================
// CONFIGURATION
// =========================================================
var SURVEY_ID = '639742'; 
var questionId = '22715';
var baseUrl = ' enquetes-dss.ens.psl.eu/ '; 
// CHANGEMENT ICI : profession_data-eng.js
var dataFilePath = baseUrl + 'upload/surveys/' + SURVEY_ID + '/files/profession_data-en.js';

var inputElement = $('#question' + questionId + ' input[type="text"]'); 
var nextButton = $('#move-next');
var minChars = 3;
var maxResults = 30;
var alertMessage = "Please indicate your profession"; 
var selectedLabel = null; 
var NOT_FOUND_CODE = "__NOT_FOUND__";

if (!inputElement.length) {
    console.error("Error: Input field not found for question " + questionId);
    return;
}

inputElement.wrap('<div style="position: relative; display: inline-block; width: 100%;"></div>');

var notFoundButtonId = 'not-found-btn-' + questionId;
// ** MODIFICATION 1 : COULEURS INITIALES DU BOUTON (background: #7748E9, color: white) **
var $notFoundButton = $('<button type="button" id="' + notFoundButtonId + '" style="margin-bottom: 10px; padding: 10px 20px; background: #7748E9; border: 2px solid #7748E9; border-radius: 5px; cursor: pointer; font-size: 14px; color: white; font-weight: 500; box-shadow: 0 2px 4px rgba(119, 72, 233, 0.4); transition: all 0.3s ease; display: none; width: auto; text-align: center;">Not found in the list</button>');
inputElement.parent().before($notFoundButton);

var $currentQuestionContainer = $('#question' + questionId).closest('[id^="question"]');
var $nextQuestionContainer = $currentQuestionContainer.nextAll('[id^="question"]').first();

// CSS pour le menu d'autocomplétion
// ** MODIFICATION 2 : COULEURS AU SURVOL DU BOUTON (:hover) + style pour .highlight **
$('<style>')
    .text('#autocomplete-menu-' + questionId + ' { position: absolute; background: white; border: 1px solid #ccc; max-height: 300px; overflow-y: auto; z-index: 10000; display: none; box-shadow: 0 2px 8px rgba(0,0,0,0.15); } ' +
          '#autocomplete-menu-' + questionId + ' .autocomplete-item { padding: 8px 12px; cursor: pointer; border-bottom: 1px solid #f0f0f0; } ' +
          '#autocomplete-menu-' + questionId + ' .autocomplete-item:hover { background: #e8f4f8; } ' +
          '#autocomplete-menu-' + questionId + ' .autocomplete-item .highlight { font-weight: bold; color: #7748E9; } ' +
          '#autocomplete-menu-' + questionId + ' .autocomplete-footer { padding: 8px 12px; background: #f5f5f5; border-top: 2px solid #ddd; font-size: 0.9em; color: #666; } ' +
          '#autocomplete-menu-' + questionId + ' .result-count { display: block; font-weight: bold; margin-bottom: 4px; color: #333; } ' +
          '#autocomplete-menu-' + questionId + ' .info-message { display: block; color: #d9534f; font-style: italic; margin-top: 4px; } ' +
          '#' + notFoundButtonId + ':hover { background: #8B68E9; border-color: #8B68E9; color: white; transform: translateY(-1px); box-shadow: 0 3px 6px rgba(119, 72, 233, 0.5); } ' +
          '#' + notFoundButtonId + ':active { transform: translateY(0); box-shadow: 0 1px 2px rgba(119, 72, 233, 0.4); }')
    .appendTo('head');

// =========================================================
// CHARGEMENT DES DONNÉES
// =========================================================
$.getScript(dataFilePath)
    .done(function() {
        if (typeof LABELS_LISTE === 'undefined') {
            console.error("ERROR: LABELS_LISTE not found in profession_data-eng.js");
            inputElement.attr('placeholder', 'Variable error').prop('disabled', true);
        } else {
            console.log("✓ profession_data-eng.js loaded (" + LABELS_LISTE.length + " labels)");
            runAutocompleteLogic();
        }
    })
    .fail(function() {
        console.error("ERROR 404: File not found - " + dataFilePath);
        inputElement.attr('placeholder', 'Network error').prop('disabled', true);
    });

// =========================================================
// AUTOCOMPLÉTION ADAPTÉE (ANGLAIS)
// =========================================================
function runAutocompleteLogic() {
    var menuId = 'autocomplete-menu-' + questionId;
    var $menu = $('<div id="' + menuId + '"></div>').insertAfter(inputElement);
    
    function normalizeString(s) {
        if (!s) return "";
        return s.toLowerCase()
                .normalize("NFD")
                .replace(/[\u0300-\u036f]/g, "")
                .trim();
    }
    
    function normalizeInclusive(s) {
        if (!s) return "";
        var normalized = normalizeString(s);
        
        // Supprimer les séparateurs d'inclusif (slash, tiret, point médian, astérisque, parenthèses)
        normalized = normalized.replace(/[·•\.\/\*]/g, '-');
        
        // Normaliser les formes inclusives anglaises
        normalized = normalized
            // Formes avec slash : he/she, his/her, him/her → forme masculine
            .replace(/\bhe\/she\b/g, 'he')
            .replace(/\bshe\/he\b/g, 'he')
            .replace(/\bhis\/her\b/g, 'his')
            .replace(/\bher\/his\b/g, 'his')
            .replace(/\bhim\/her\b/g, 'him')
            .replace(/\bher\/him\b/g, 'him')
            .replace(/\bs\/he\b/g, 'he')
            
            // Formes avec parenthèses : actor(ess), waiter(ess) → forme masculine
            .replace(/\(ess\)/g, '')                          // actor(ess) → actor
            .replace(/\(trix\)/g, 'tor')                      // aviator(trix) → aviator
            .replace(/\(woman\)/g, 'man')                     // spokesman(woman) → spokesman
            
            // Formes avec slash pour les métiers : actor/actress, waiter/waitress → forme masculine
            .replace(/or\/ess\b/g, 'or')                      // actor/ess → actor
            .replace(/or\/ress\b/g, 'or')                     // waiter/ress → waiter
            .replace(/man\/woman\b/g, 'man')                  // spokesman/woman → spokesman
            
            // Formes avec tiret : actor-ess, waiter-ress → forme masculine
            .replace(/or-ess\b/g, 'or')                       // actor-ess → actor
            .replace(/or-ress\b/g, 'or')                      // waiter-ress → waiter
            .replace(/man-woman\b/g, 'man')                   // spokesman-woman → spokesman
            
            // Formes neutres avec x : latinx, folx → forme standard
            .replace(/\blatinx\b/g, 'latino')
            .replace(/\bfolx\b/g, 'folks')
            .replace(/x(es)?\b/g, '')                         // womxn → womn (puis traité plus bas)
            
            // Formes avec astérisque : actor*, waiter* → forme masculine
            .replace(/\*\b/g, '')                             // actor* → actor
            
            // Normaliser les formes féminines vers masculin/neutre
            .replace(/\bactress(es)?\b/g, 'actor')            // actress → actor
            .replace(/\bwaitress(es)?\b/g, 'waiter')          // waitress → waiter
            .replace(/\bhostess(es)?\b/g, 'host')             // hostess → host
            .replace(/\bstewardess(es)?\b/g, 'steward')       // stewardess → steward
            .replace(/\baviatrix\b/g, 'aviator')              // aviatrix → aviator
            .replace(/\bexecutrix\b/g, 'executor')            // executrix → executor
            .replace(/\badministratrix\b/g, 'administrator')  // administratrix → administrator
            .replace(/\bspokeswoman\b/g, 'spokesman')         // spokeswoman → spokesman
            .replace(/\bbusinesswoman\b/g, 'businessman')     // businesswoman → businessman
            .replace(/\bchairwoman\b/g, 'chairman')           // chairwoman → chairman
            .replace(/\bcongresswoman\b/g, 'congressman')     // congresswoman → congressman
            .replace(/\bpolicewoman\b/g, 'policeman')         // policewoman → policeman
            .replace(/\bfirewoman\b/g, 'fireman')             // firewoman → fireman
            .replace(/\bsaleswoman\b/g, 'salesman')           // saleswoman → salesman
            .replace(/\bcameraman\/woman\b/g, 'cameraman')    // cameraman/woman → cameraman
            .replace(/\bwoman\b/g, 'man')                     // woman → man (pour les composés restants)
            
            // Formes neutres they/them (garder tel quel car déjà neutre)
            // Ces formes sont déjà inclusives et ne nécessitent pas de normalisation
            
            // Nettoyer les doubles tirets ou espaces créés
            .replace(/-+/g, '-')
            .replace(/\s+/g, ' ')
            .trim();
        
        return normalized;
    }
    
    function normalizeForSearch(s) {
        if (!s) return "";
        var normalized = normalizeInclusive(s);
        return normalized.replace(/[\.\/\-]/g, ' ');
    }

    // Nouvelle fonction pour surligner les correspondances
    function highlightMatches(originalLabel, searchInput) {
        var normalizedInput = normalizeInclusive(searchInput);
        var searchWords = normalizedInput.split(/\s+/).filter(function(w) { return w.length > 0; });
        
        if (searchWords.length === 0) {
            return originalLabel;
        }
        
        var result = originalLabel;
        
        // Pour chaque mot de recherche, trouver les correspondances dans le label
        for (var w = 0; w < searchWords.length; w++) {
            var searchWord = searchWords[w];
            if (searchWord.length === 0) continue;
            
            // Diviser le label en mots pour les analyser
            var words = result.split(/(\s+|[^\wÀ-ÿ]+)/);
            var newWords = [];
            
            for (var i = 0; i < words.length; i++) {
                var word = words;
                
                // Si c'est un séparateur ou déjà surligné, le garder tel quel
                if (!word || word.match(/^\s+$/) || word.match(/^[^\wÀ-ÿ]+$/) || word.indexOf('<span') >= 0) {
                    newWords.push(word);
                    continue;
                }
                
                // Normaliser le mot pour la comparaison
                var normalizedWord = normalizeInclusive(word);
                
                // Si le mot commence par le mot recherché, le surligner
                if (normalizedWord.startsWith(searchWord) || normalizedWord.indexOf(searchWord) === 0) {
                    // Déterminer la longueur à surligner en tenant compte de la normalisation
                    var highlightLength = searchWord.length;
                    
                    // Ajuster pour les caractères qui ont été normalisés
                    var charsToHighlight = 0;
                    var normalizedCount = 0;
                    
                    for (var c = 0; c < word.length && normalizedCount < highlightLength; c++) {
                        charsToHighlight++;
                        var normalizedChar = normalizeInclusive(word[c]);
                        normalizedCount += normalizedChar.length;
                    }
                    
                    var highlighted = '<span class="highlight">' + word.substring(0, charsToHighlight) + '</span>' + word.substring(charsToHighlight);
                    newWords.push(highlighted);
                } else {
                    newWords.push(word);
                }
            }
            
            result = newWords.join('');
        }
        
        return result;
    }

    function searchLabels(searchInput) {
        var normalizedInput = normalizeInclusive(searchInput);
        // CORRECTION: Utiliser le même pattern de split que pour les labels ([^a-z0-9]+)
        // Cela permet de gérer correctement les apostrophes et autres caractères spéciaux
        var searchWords = normalizedInput.split(/[^a-z0-9]+/).filter(function(w) { return w.length > 0; });
        var matchingLabels = [];

        if (searchInput.length < minChars) return { results: [], count: 0, message: "", searchInput: searchInput };
        if (searchWords.length === 0) return { results: [], count: 0, message: "Search terms are too short", searchInput: searchInput };

        for (var i = 0; i < LABELS_LISTE.length; i++) {
            var originalLabel = LABELS_LISTE;
            var normalizedLabel = normalizeInclusive(originalLabel);
            var significantWords = normalizedLabel.split(/[^a-z0-9]+/).filter(function(w) { return w.length > 0; });
            
            var allWordsMatch = true;
            for (var k = 0; k < searchWords.length; k++) {
                var searchWord = searchWords[k];
                var foundMatch = false;
                
                for (var j = 0; j < significantWords.length; j++) {
                    if (significantWords[j].startsWith(searchWord)) {
                        foundMatch = true;
                        break;
                    }
                }
                
                if (!foundMatch) {
                    allWordsMatch = false;
                    break;
                }
            }
            
            if (allWordsMatch) matchingLabels.push(originalLabel);
        }

        matchingLabels.sort(function(a, b) {
            var normA = normalizeInclusive(a);
            var normB = normalizeInclusive(b);
            
            var startsWithA = normA.startsWith(normalizedInput);
            var startsWithB = normB.startsWith(normalizedInput);
            
            if (startsWithA !== startsWithB) {
                return startsWithB - startsWithA;
            }
            
            var firstSearchWord = searchWords[0];
            var wordsA = normA.split(/[^a-z0-9]+/).filter(function(w) { return w.length > 0; });
            var wordsB = normB.split(/[^a-z0-9]+/).filter(function(w) { return w.length > 0; });
            var firstWordA = wordsA[0] || '';
            var firstWordB = wordsB[0] || '';
            var firstMatchA = firstWordA.startsWith(firstSearchWord);
            var firstMatchB = firstWordB.startsWith(firstSearchWord);
            
            if (firstMatchA !== firstMatchB) {
                return firstMatchB - firstMatchA;
            }
            
            return normA.localeCompare(normB);
        });

        var resultCount = matchingLabels.length;
        var message = "";
        if (resultCount === 0) {
            message = "No results found; please refine your search or use the button below.";
        } else if (resultCount > maxResults) {
            message = "Too many results, please keep typing...";
        }

        return { 
            results: matchingLabels.slice(0, maxResults), 
            count: resultCount, 
            message: message,
            searchInput: searchInput
        };
    }
    
    function showMenu(searchData) {
        var html = '<div class="autocomplete-footer">';
        html += '<span class="result-count">Results found: ' + searchData.count + '</span>';
        if (searchData.message) html += '<span class="info-message">' + searchData.message + '</span>';
        html += '</div>';
        
        // Ajouter les résultats avec le texte surligné
        for (var i = 0; i < searchData.results.length; i++) {
            var originalLabel = searchData.results;
            var highlightedLabel = highlightMatches(originalLabel, searchData.searchInput);
            
            html += '<div class="autocomplete-item" data-value="' + 
                    originalLabel.replace(/"/g, '&quot;') + '">' + 
                    highlightedLabel + '</div>';
        }
        
        $menu.html(html).show();
        $menu.css({
            top: inputElement.outerHeight() + 'px',
            width: inputElement.outerWidth() + 'px'
        });
    }

    // CACHE la question suivante par défaut au chargement
    if ($nextQuestionContainer && $nextQuestionContainer.length) {
        $nextQuestionContainer.hide();
    }

    var searchTimeout;
    inputElement.on('input', function() {
        selectedLabel = null;
        if ($nextQuestionContainer) $nextQuestionContainer.hide();
        var value = $(this).val().trim();
        
        clearTimeout(searchTimeout);
        if (value.length < minChars) {
            $menu.hide();
            $('#' + notFoundButtonId).hide();
            return;
        }
        
        searchTimeout = setTimeout(function() {
            var data = searchLabels(value);
            if (data.results.length > 0 || data.message) {
                showMenu(data);
                $('#' + notFoundButtonId).show();
            }
        }, 150);
    });

    $menu.on('click', '.autocomplete-item', function() {
        var val = $(this).attr('data-value');
        selectedLabel = val;
        inputElement.val(val);
        $menu.hide();
    });

    $('#' + notFoundButtonId).on('click', function() {
        selectedLabel = NOT_FOUND_CODE;
        inputElement.val("Not found in the list");
        $menu.hide();
        if ($nextQuestionContainer) $nextQuestionContainer.show();
    });

    nextButton.on('click.validation', function(e) {
        if (inputElement.val().trim().length > 0 && selectedLabel === null) {
            e.preventDefault();
            alert(alertMessage);
        }
    });

    console.log("✓ Custom English autocomplete initialized");
}I am concerned about potential performance issues, such as browser lag or technical glitches, when dozens or hundreds of users access these questions simultaneously.How can I ensure that this setup remains stable and responsive under load with such a large dataset? Are there specific optimizations you would recommend for handling 5,000+ items within LimeSurvey?Thank you very much for your help.Best regards,Louuuu

Please Log in to join the conversation.

More
2 weeks 6 days ago - 2 weeks 6 days ago #273721 by Joffm
Replied by Joffm on topic Performance concerns: Custom JS searchable dropdown with external resource file
Hi,
if you enter code, please use the button to insert a well formatted code
 

Nevertheless, you'd better send the lss export of these relevant questions. (copy the survey, remove everything not related, and send lss export of the rest)

And I wonder why you didn't use the - more or less - standard in LimeSurvey to insert an "autocomplete"
1. The plugin (here I have to admit that it is really really slow when there are more then 500 entries)
2. The  solution with the library "/jquery.csv.min.js".
 

There are more than 9000 zip codes in the file and it works without noticable delay.

It's only this code
Code:
<link href="/lime6/upload/surveys/{SID}/files/jquery-ui.min.css" rel="stylesheet" />
<script src="/lime6/upload/surveys/{SID}/files/jquery-ui.min.js"></script>
<script src="/lime6/upload/surveys/{SID}/files/jquery.csv.min.js"></script>
<script type="text/javascript" charset="utf-8">
  $(document).on('ready pjax:complete',function() {
    var url = "/lime6/upload/surveys/{SID}/files/plzort.csv";
    var PLZ = new Array();
 
    $.get(url,function(data){
      fullArray = $.csv.toArrays(data);
      $(fullArray).each(function(i, item){
        PLZ.push(item[0]);
      });
      $("#question{QID} input[type=text]").autocomplete({
        minLength: 2,
        source: PLZ
      });
    });
  });
</script>


Unfortunately we do not know which question type you want to use - a simple dropdown, or an "array(text)" or dropdowns in a "multiple short text" question, you'd send the lss export.

BTW: Here the example with your file.
 
 
Best regards
Joffm

It is also possible to use the "array" solution, as your file is rather small.
You find all this here in the manual:
[url] www.limesurvey.org/manual/Workarounds:_M...for_LimeSurvey_2.05+ [/url]
 
Volunteers are not paid.
Not because they are worthless, but because they are priceless
Last edit: 2 weeks 6 days ago by Joffm.

Please Log in to join the conversation.

More
2 weeks 4 days ago - 2 weeks 4 days ago #273743 by Joffm
Replied by Joffm on topic Performance concerns: Custom JS searchable dropdown with external resource file
And there is another library "easy-autocomplete".
As I wrote before, you can insert your whole file into the question text. The database field is of type "mediumtext"

Can look like this; a question of type "short free text" 
Code:
<script src="https://cdnjs.cloudflare.com/ajax/libs/easy-autocomplete/1.3.5/jquery.easy-autocomplete.min.js"></script>
<link href="https://cdnjs.cloudflare.com/ajax/libs/easy-autocomplete/1.3.5/easy-autocomplete.css" rel="stylesheet" />
 
<script type="text/javascript">
  $(document).on('ready pjax:scriptcomplete',function(){
 
    // Define the data
       var options = {
       data: [
        "Lampshade maker",
        "Tree feller / Logger",
        "Slaughterer",
        "Quarry worker",
...
        "Escort",
        "Dealer",
        "Drug dealer"
      ],
      list: {
        maxNumberOfElements: 100,
        match: {
          enabled: true
        }
      }
    };
 
    $("#question{QID} input[type=text]").easyAutocomplete(options);
  });
</script>


And some css to get the scrollbars, the background color, and something else
Code:
<style type="text/css">
.easy-autocomplete-container {
  position: relative;
  max-height: 300px;
  overflow: auto;
}
.easy-autocomplete-container ul {
  background-color: Gainsboro;
  }
@media (min-width: 481px) {
  .ls-input-group-extra {
    vertical-align: top;
  }
}
</style>

 

But you should revise the file.
Several professions are doubled or even tripled (not only this "goat farmer/breeder")
 

Joffm

But there is still the request to show your layout of the question (lss Export)
 
Volunteers are not paid.
Not because they are worthless, but because they are priceless
Last edit: 2 weeks 4 days ago by Joffm.

Please Log in to join the conversation.

Moderators: tpartner, holch

Lime-years ahead

Online-surveys for every purse and purpose

Pricing & Plans
Get started

Legal

About Us

Open Source