|
|
<!DOCTYPE html> |
|
|
|
|
|
<html lang="en" data-content_root="./"> |
|
|
<head> |
|
|
<meta charset="utf-8" /> |
|
|
<meta name="viewport" content="width=device-width, initial-scale=1.0" /><meta name="viewport" content="width=device-width, initial-scale=1" /> |
|
|
|
|
|
<title>AIM-CU documentation — AIM-CU 1.0.0 documentation</title> |
|
|
<link rel="stylesheet" type="text/css" href="_static/pygments.css?v=5ecbeea2" /> |
|
|
<link rel="stylesheet" type="text/css" href="_static/basic.css?v=d35aba6c" /> |
|
|
<link rel="stylesheet" type="text/css" href="_static/alabaster.css?v=0ca6144b" /> |
|
|
<script src="_static/documentation_options.js?v=8d563738"></script> |
|
|
<script src="_static/doctools.js?v=9bcbadda"></script> |
|
|
<script src="_static/sphinx_highlight.js?v=dc90522c"></script> |
|
|
<script async="async" src="https://cdn.jsdelivr.net/npm/mathjax@3/es5/tex-mml-chtml.js"></script> |
|
|
<link rel="index" title="Index" href="genindex.html" /> |
|
|
<link rel="search" title="Search" href="search.html" /> |
|
|
<link rel="next" title="Methods" href="ref_method.html" /> |
|
|
|
|
|
<link rel="stylesheet" href="_static/custom.css" type="text/css" /> |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
</head><body> |
|
|
|
|
|
|
|
|
<div class="document"> |
|
|
<div class="documentwrapper"> |
|
|
<div class="bodywrapper"> |
|
|
|
|
|
|
|
|
<div class="body" role="main"> |
|
|
|
|
|
<section id="aim-cu-documentation"> |
|
|
<h1>AIM-CU documentation<a class="headerlink" href="#aim-cu-documentation" title="Link to this heading">¶</a></h1> |
|
|
<p>A CUSUM-based tool for AI Monitoring</p> |
|
|
<p>Monitoring a clinically deployed AI device to detect performance drift is an essential step to ensure the safety and effectiveness of AI.</p> |
|
|
<p>AIM-CU is a statistical tool for AI monitoring using cumulative sum (AIM-CU).</p> |
|
|
<p>AIM-CU computes:</p> |
|
|
<ul class="simple"> |
|
|
<li><p>The parameter choices for change-point detection based on an acceptable false alarm rate</p></li> |
|
|
<li><p>Detection delay estimates for a given displacement of the performance metric from the target for those parameter choices.</p></li> |
|
|
</ul> |
|
|
<section id="system-setup"> |
|
|
<h2>System setup<a class="headerlink" href="#system-setup" title="Link to this heading">¶</a></h2> |
|
|
<p>Make sure R is installed in the system. Instructions for linux (the below setup is only performed in linux):</p> |
|
|
<div class="highlight-shell notranslate"><div class="highlight"><pre><span></span>wget<span class="w"> </span>-qO-<span class="w"> </span>https://cloud.r-project.org/bin/linux/ubuntu/marutter_pubkey.asc<span class="w"> </span><span class="p">|</span><span class="w"> </span>tee<span class="w"> </span>-a<span class="w"> </span>/etc/apt/trusted.gpg.d/cran_ubuntu_key.asc |
|
|
add-apt-repository<span class="w"> </span><span class="s2">"deb https://cloud.r-project.org/bin/linux/ubuntu </span><span class="k">$(</span>lsb_release<span class="w"> </span>-cs<span class="k">)</span><span class="s2">-cran40/"</span> |
|
|
apt-get<span class="w"> </span>install<span class="w"> </span>-y<span class="w"> </span>--no-install-recommends<span class="w"> </span>r-base<span class="w"> </span>r-base-dev |
|
|
|
|
|
<span class="c1"># setup R configs</span> |
|
|
<span class="nb">echo</span><span class="w"> </span><span class="s2">"r <- getOption('repos'); r['CRAN'] <- 'http://cran.us.r-project.org'; options(repos = r);"</span><span class="w"> </span>><span class="w"> </span>~/.Rprofile |
|
|
Rscript<span class="w"> </span>-e<span class="w"> </span><span class="s2">"install.packages('ggplot2')"</span> |
|
|
Rscript<span class="w"> </span>-e<span class="w"> </span><span class="s2">"install.packages('hexbin')"</span> |
|
|
Rscript<span class="w"> </span>-e<span class="w"> </span><span class="s2">"install.packages('lazyeval')"</span> |
|
|
Rscript<span class="w"> </span>-e<span class="w"> </span><span class="s2">"install.packages('cusumcharter')"</span> |
|
|
Rscript<span class="w"> </span>-e<span class="w"> </span><span class="s2">"install.packages('RcppCNPy')"</span> |
|
|
Rscript<span class="w"> </span>-e<span class="w"> </span><span class="s2">"install.packages('spc')"</span> |
|
|
</pre></div> |
|
|
</div> |
|
|
</section> |
|
|
<section id="code-execution"> |
|
|
<h2>Code execution<a class="headerlink" href="#code-execution" title="Link to this heading">¶</a></h2> |
|
|
<p>Clone AIM-CU repository.</p> |
|
|
<div class="highlight-shell notranslate"><div class="highlight"><pre><span></span>git<span class="w"> </span>clone<span class="w"> </span>https://github.com/DIDSR/AIM-CU.git |
|
|
</pre></div> |
|
|
</div> |
|
|
<p>Run the following commands to install required dependencies (Python = 3.10 is used).</p> |
|
|
<div class="highlight-shell notranslate"><div class="highlight"><pre><span></span>apt-get<span class="w"> </span>-y<span class="w"> </span>install<span class="w"> </span>python3 |
|
|
apt-get<span class="w"> </span>-y<span class="w"> </span>install<span class="w"> </span>pip |
|
|
<span class="nb">cd</span><span class="w"> </span>AIM-CU |
|
|
pip<span class="w"> </span>install<span class="w"> </span>-r<span class="w"> </span>requirements.txt |
|
|
</pre></div> |
|
|
</div> |
|
|
<p>Run AIM-CU.</p> |
|
|
<div class="highlight-shell notranslate"><div class="highlight"><pre><span></span><span class="nb">cd</span><span class="w"> </span>src/package |
|
|
python3<span class="w"> </span>app.py |
|
|
</pre></div> |
|
|
</div> |
|
|
<p>Open the URL <a class="reference external" href="http://0.0.0.0:7860">http://0.0.0.0:7860</a> that is running the AIM-CU locally.</p> |
|
|
</section> |
|
|
<section id="example-code-execution"> |
|
|
<h2>Example code execution<a class="headerlink" href="#example-code-execution" title="Link to this heading">¶</a></h2> |
|
|
<p>Example code can be run in a Jupyter Notebook after opening it with <code class="docutils literal notranslate"><span class="pre">jupyter</span> <span class="pre">notebook</span></code> command from <code class="docutils literal notranslate"><span class="pre">/src/package/</span></code> directory. The tool is designed to used through UI, not from console.</p> |
|
|
</section> |
|
|
<section id="demo"> |
|
|
<h2>Demo<a class="headerlink" href="#demo" title="Link to this heading">¶</a></h2> |
|
|
<p>AIM-CU can also be run through the demo available at <a class="reference external" href="https://huggingface.co/spaces/didsr/AIM-CU">https://huggingface.co/spaces/didsr/AIM-CU</a>. If Space is paused, click on Restart button.</p> |
|
|
</section> |
|
|
<section id="usability"> |
|
|
<h2>Usability<a class="headerlink" href="#usability" title="Link to this heading">¶</a></h2> |
|
|
<ul class="simple"> |
|
|
<li><p>Example AI output CSV file is available as <a class="reference external" href="config/spec-60-60.csv">config/spec-60-60.csv</a> to be uploaded in monitoring phase.</p></li> |
|
|
<li><p>Workflow instruction to run the tool is available at bottom-left of UI.</p></li> |
|
|
<li><p>Sample UI output is available at <a class="reference external" href="assets/ui.png">assets/ui.png</a>.</p></li> |
|
|
<li><p>Setting <code class="docutils literal notranslate"><span class="pre">control:save_figure</span></code> to <code class="docutils literal notranslate"><span class="pre">true</span></code> from <a class="reference external" href="config/config.toml">config.toml</a> will save tables and plots in <a class="reference external" href="figure/">figure/</a>.</p></li> |
|
|
<li><p>Running AIM-CU does not take time longer than a few seconds, and it does not require GPU.</p></li> |
|
|
</ul> |
|
|
</section> |
|
|
<section id="aim-cu"> |
|
|
<h2>AIM-CU<a class="headerlink" href="#aim-cu" title="Link to this heading">¶</a></h2> |
|
|
<div class="toctree-wrapper compound"> |
|
|
<p class="caption" role="heading"><span class="caption-text">Contents:</span></p> |
|
|
<ul> |
|
|
<li class="toctree-l1"><a class="reference internal" href="ref_method.html">Methods</a><ul> |
|
|
<li class="toctree-l2"><a class="reference internal" href="ref_method.html#cusum-parameters">CUSUM parameters</a></li> |
|
|
<li class="toctree-l2"><a class="reference internal" href="ref_method.html#cusum-chart">CUSUM chart</a></li> |
|
|
</ul> |
|
|
</li> |
|
|
<li class="toctree-l1"><a class="reference internal" href="ref_cusum.html">CUSUM</a><ul> |
|
|
<li class="toctree-l2"><a class="reference internal" href="ref_cusum.html#package.cusum.CUSUM"><code class="docutils literal notranslate"><span class="pre">CUSUM</span></code></a></li> |
|
|
</ul> |
|
|
</li> |
|
|
<li class="toctree-l1"><a class="reference internal" href="ref_theoretical.html">ARLTheoretical</a><ul> |
|
|
<li class="toctree-l2"><a class="reference internal" href="ref_theoretical.html#package.ARLTheoretical.get_ARL_1"><code class="docutils literal notranslate"><span class="pre">get_ARL_1()</span></code></a></li> |
|
|
<li class="toctree-l2"><a class="reference internal" href="ref_theoretical.html#package.ARLTheoretical.get_ARL_1_h_mu1_k"><code class="docutils literal notranslate"><span class="pre">get_ARL_1_h_mu1_k()</span></code></a></li> |
|
|
<li class="toctree-l2"><a class="reference internal" href="ref_theoretical.html#package.ARLTheoretical.get_ref_value"><code class="docutils literal notranslate"><span class="pre">get_ref_value()</span></code></a></li> |
|
|
<li class="toctree-l2"><a class="reference internal" href="ref_theoretical.html#package.ARLTheoretical.get_ref_value_k"><code class="docutils literal notranslate"><span class="pre">get_ref_value_k()</span></code></a></li> |
|
|
</ul> |
|
|
</li> |
|
|
<li class="toctree-l1"><a class="reference internal" href="ref_utils.html">Utils</a><ul> |
|
|
<li class="toctree-l2"><a class="reference internal" href="ref_utils.html#package.utils.get_greattable_as_html"><code class="docutils literal notranslate"><span class="pre">get_greattable_as_html()</span></code></a></li> |
|
|
<li class="toctree-l2"><a class="reference internal" href="ref_utils.html#package.utils.populate_summary_table_ARL0_k"><code class="docutils literal notranslate"><span class="pre">populate_summary_table_ARL0_k()</span></code></a></li> |
|
|
<li class="toctree-l2"><a class="reference internal" href="ref_utils.html#package.utils.populate_summary_table_ARL1_k"><code class="docutils literal notranslate"><span class="pre">populate_summary_table_ARL1_k()</span></code></a></li> |
|
|
</ul> |
|
|
</li> |
|
|
</ul> |
|
|
</div> |
|
|
</section> |
|
|
<section id="disclaimer"> |
|
|
<h2>Disclaimer<a class="headerlink" href="#disclaimer" title="Link to this heading">¶</a></h2> |
|
|
<p>This software and documentation was developed at the Food and Drug Administration (FDA) by employees of the Federal Government in the course of their official duties. Pursuant to Title 17, Section 105 of the United States Code, this work is not subject to copyright protection and is in the public domain. Permission is hereby granted, free of charge, to any person obtaining a copy of the Software, to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, or sell copies of the Software or derivatives, and to permit persons to whom the Software is furnished to do so. FDA assumes no responsibility whatsoever for use by other parties of the Software, its source code, documentation or compiled executables, and makes no guarantees, expressed or implied, about its quality, reliability, or any other characteristic. Further, use of this code in no way implies endorsement by the FDA or confers any advantage in regulatory decisions. Although this software can be redistributed and/or modified freely, we ask that any derivative works bear some notice that they are derived from it, and any modified versions bear some notice that they have been modified.</p> |
|
|
</section> |
|
|
</section> |
|
|
|
|
|
|
|
|
</div> |
|
|
|
|
|
</div> |
|
|
</div> |
|
|
<div class="sphinxsidebar" role="navigation" aria-label="Main"> |
|
|
<div class="sphinxsidebarwrapper"> |
|
|
<h1 class="logo"><a href="#">AIM-CU</a></h1> |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
<p> |
|
|
<iframe src="https://ghbtns.com/github-btn.html?user=DIDSR&repo=AIM-CU&type=watch&count=true&size=large&v=2" |
|
|
allowtransparency="true" frameborder="0" scrolling="0" width="200px" height="35px"></iframe> |
|
|
</p> |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
<search id="searchbox" style="display: none" role="search"> |
|
|
<div class="searchformwrapper"> |
|
|
<form class="search" action="search.html" method="get"> |
|
|
<input type="text" name="q" aria-labelledby="searchlabel" autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false" placeholder="Search"/> |
|
|
<input type="submit" value="Go" /> |
|
|
</form> |
|
|
</div> |
|
|
</search> |
|
|
<script>document.getElementById('searchbox').style.display = "block"</script><h3>Navigation</h3> |
|
|
<p class="caption" role="heading"><span class="caption-text">Contents:</span></p> |
|
|
<ul> |
|
|
<li class="toctree-l1"><a class="reference internal" href="ref_method.html">Methods</a></li> |
|
|
<li class="toctree-l1"><a class="reference internal" href="ref_cusum.html">CUSUM</a></li> |
|
|
<li class="toctree-l1"><a class="reference internal" href="ref_theoretical.html">ARLTheoretical</a></li> |
|
|
<li class="toctree-l1"><a class="reference internal" href="ref_utils.html">Utils</a></li> |
|
|
</ul> |
|
|
|
|
|
<div class="relations"> |
|
|
<h3>Related Topics</h3> |
|
|
<ul> |
|
|
<li><a href="#">Documentation overview</a><ul> |
|
|
<li>Next: <a href="ref_method.html" title="next chapter">Methods</a></li> |
|
|
</ul></li> |
|
|
</ul> |
|
|
</div> |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
</div> |
|
|
</div> |
|
|
<div class="clearer"></div> |
|
|
</div> |
|
|
<div class="footer"> |
|
|
|
|
|
|
|
|
|
|
|
Powered by <a href="https://www.sphinx-doc.org/">Sphinx 8.1.3</a> |
|
|
& <a href="https://alabaster.readthedocs.io">Alabaster 1.0.0</a> |
|
|
|
|
|
| |
|
|
<a href="_sources/index.rst.txt" |
|
|
rel="nofollow">Page source</a> |
|
|
</div> |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
</body> |
|
|
</html> |
