You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: Azure/README.md
+40-23Lines changed: 40 additions & 23 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -16,46 +16,68 @@ Follow the steps highlighted [here](https://github.com/NIGMS/NIGMS-Sandbox/blob/
16
16
The Machine Learning Studio Jupyter Notebooks already have an R kernel available which we will use and install our packages to.
17
17
18
18
19
-
Follow the steps highlighted in part two (2. Spin up Instance from a Container) of [here](https://github.com/NIGMS/NIGMS-Sandbox/blob/main/docs/HowToCreateAzureMLNotebooks.md) to create a new notebook instance in Machine Learning Studio. Follow steps 1-8, in step 5 select region us-east4 (Northern Virginia) and be especially careful to use custom container `us-east4-docker.pkg.dev/nih-cl-shared-resources/nigms-sandbox/nigms-vertex-r` in step 6 under the Docker container image prompt. In step 7 under the Machine type tab, select n1-standard-4 from the dropdown box. In step 8, be careful to **Enable Idle Shutdown**. After creating the notebook, you can click on **OPEN JUPYTERLAB**.
19
+
Follow the steps highlighted [here](https://github.com/NIGMS/NIGMS-Sandbox/blob/main/docs/HowToCreateAzureMLNotebooks.md) to create a Azure Machine Learning workspace and a new notebook instance that workspace. Follow steps 1-6, when creating your instance. Idle shutdown should automatically be enabled for 1 hour. You have two options to interact with these tutorials:
20
+
21
+
Option 1 uses the 'Notebook' tab in the left-hand menu to view an Azure UI version of interacting with notebooks. If you are using option 1 after navigating to **Notebooks**, start your instance, and then you can run your notebooks.
22
+
23
+
Option 2 allows you to enter in a JupyterLab setting to interact with notebooks. If you are using option 2 after creating the instance to run the notebooks under 'Compute', you can click **Start**. Then under 'Applications' click **JUPYTERLAB**.
20
24
21
25
### Downloading and Running Tutorial Files
22
26
23
-
Now that you have successfully created your virtual machine, you will be directed to Jupyterlab screen. The next step is to import the notebooks and start the course.
24
-
This can be done by selecting the __Git__ from the top menu in Jupyterlab, and choosing the __Clone a Repository__
27
+
Now that you have successfully created your virtual machine, you can use either the Azure Notebook UI screen or Jupyterlab screen to interact with the tutorials. The next step is to import the notebooks and start the course.
28
+
29
+
This can be done in either two ways:
30
+
31
+
**Azure Notebook UI Setting**
32
+
Open the terminal, type in `git clone https://github.com/NIGMS/Consensus-Pathway-Analysis-in-the-Cloud.git`, then hit enter.
33
+
34
+
35
+
36
+
**JupyterLab Setting**
37
+
Selecting the __Git__ from the top menu in Jupyterlab, and choosing the __Clone a Repository__
25
38
option. Next, you can copy and paste in the link of the repository: `https://github.com/NIGMS/Consensus-Pathway-Analysis-in-the-Cloud.git` and click __Clone__.
26
39
27
40
28
41
29
-
This will download the repository to your JupyterLab folder. All tutorial files for the five submodules are in Jupyter format with a .ipynb extension. Double-click each file to view the lab content and run the code. This will open the Jupyter file in a Jupyter notebook. From here, you can run each section, or "cell", of the code, one by one, by pressing the "Play" button in the menu above.
42
+
This will download the repository to your JupyterLab folder. All tutorial files for the five submodules are in Jupyter format with a .ipynb extension. Double-click each file to view its content and select your kernel in the top left corner to run the notebook in a specific conda environment (e.g., R, Python, etc.). From here, you can run each section, or "cell", of the code, one by one, by pressing the "Play" button next to the cell for the Azure Notebook UI or above for the JupyterLab setting.
43
+
44
+
**Azure Notebook UI Setting**
45
+
46
+
47
+
48
+
**JupyterLab Setting**
30
49
31
50
32
51
33
-
Some 'cells' of code take longer for the computer to process than others. You will know a cell is running when a cell has an asterisk next to it \[\*\]. When the cell finishes running, that asterisk will be replaced with a number which represents the order that cell was run in. You can now explore the tutorials by running the code in each, from top to bottom. Look at the 'workflows' section below for a short description of each tutorial.
52
+
Some 'cells' of code take longer for the computer to process than others. You will know a cell is running when a cell has an asterisk next to it \[\*\]. When the cell finishes running, that asterisk will be replaced with a number, which represents the order the cell was run. You can now explore the tutorials by running the code in each, from top to bottom. Look at the 'workflows' section below for a short description of each tutorial.
53
+
54
+
Jupyter is a powerful tool with many useful features. For more information on how to use Jupyter, we recommend searching for Jupyter tutorials and literature online.
34
55
35
-
Jupyter is a powerful tool, with many useful features. For more information on how to use Jupyter, we recommend
36
-
searching for Jupyter tutorials and literature online.
56
+
### Stopping Your Instance/Virtual Machine
37
57
38
-
### Stopping Your Virtual Machine
58
+
When you are finished running code, you should turn off your notebook by turning off the instance attached to your notebook to prevent unnecessary billing or resource use by navigating to **Compute** in the side menu, selecting your instance, then clicking the __STOP__ button.
39
59
40
-
When you are finished running code, you should turn off your notebook to prevent unnecessary billing or resource use by checking your notebook and pushing the __STOP__ button.
60
+
41
61
42
62
## Creating Azure Blob Storage
43
63
In this section, we will describe the steps to create Azure Blob Storage to store data generated during analysis. The storage can be created via GUI or using the command line.
44
64
45
65
Azure Storage is comprised of three parts: accounts, containers, and blobs, as seen in the image below. Accounts contain containers which act as folders. Containers hold files, aka blobs and folders or subfolders.
When you create the Azure Machine Learning service, it will automatically create a storage account under the name of the workspace you created before. This allows you to either use the previously made storage account or create a new one. Instructions for both options are outlined below.
70
+
50
71
**Option 1:Using a Premade Storage Accounts**
51
72
1. On the webpage of your Azure account, find and select `Azure Storage Accounts`.
52
73
2. Find and click the storage account that is labeled the same as your Machine learning service.
4. Navigate to `Containers` on the left side menu.
55
76
5. The console will list all the containers and files created as shown in the figure below. Some of these containers and files are automatically created when we create our Machine Learning Service:
7. Click on the new container, then click `Upload` to upload files.
60
82
61
83
@@ -91,15 +113,10 @@ You can learn more about `az storage` commands by reading the article [here](htt
91
113
## Azure Architecture
92
114
93
115
# 
94
-
The figure above shows the architecture of the learning module with Azure infrastructure. First, we will create
95
-
an Machine Learning Studio Jupyter Notebook with R kernel. The code and instructions for each submodule are presented in a separate Jupyter Notebook.
96
-
User can either upload the Notebooks to the Machine Learning Studio Notebooks or clone from the project repository. Then, users can execute
97
-
the code directly in the Notebook. In our learning course, the submodule 01 will download data from the public repository (e.g., GEO database)
98
-
for preprocessing and save the processed data to a local file in Machine Learning Studio and to the user's Azure Storage Container. The output
99
-
of the submodule 01 will be used as inputs for all other submodules. The outputs of the submodules 02, 03, and 04 will be saved to
100
-
local repository in Machine Learning Studio Notebooks and the code to copy them to the user's cloud storage is also included.
116
+
The figure above shows the architecture of the learning module with Azure infrastructure. First, we will create a Machine Learning Studio Jupyter Notebook with the premade R kernel. The code and instructions for each submodule are presented in a separate Jupyter Notebook.
117
+
The user can either upload the Notebooks to the Machine Learning Studio Notebooks or clone from the project repository. Then, users can execute
118
+
the code directly in the Notebook. In our learning course, the submodule 01 will download data from the public repository (e.g., the GEO database)
119
+
for preprocessing and then save the processed data to a local file in the notebook and to the user's Azure Storage Container. The output
120
+
of the submodule 01 will be used as the input for all other submodules. The outputs of the submodules 02, 03, and 04 will be saved to the
121
+
local repository in Machine Learning Studio Notebooks, and the code to copy them to the user's cloud storage is also included.
Copy file name to clipboardExpand all lines: Azure/Submodule01-ProcessingExpressionData.ipynb
+18-17Lines changed: 18 additions & 17 deletions
Original file line number
Diff line number
Diff line change
@@ -108,32 +108,40 @@
108
108
"To run this tutorial, let's first download all of the packages needed. For Jupyter Notebooks in Azure Machine Learning Studio, we must specify our library as `/home/azureuser` otherwise we can run into errors."
109
109
]
110
110
},
111
+
{
112
+
"cell_type": "markdown",
113
+
"id": "9d4c35b6-56d6-4b08-b612-59cf31244818",
114
+
"metadata": {},
115
+
"source": [
116
+
"Add our library to our path to easily load our packages."
"Alternately, users can also upload their data to Azure Storage. The data may be lost after users delete the Machine Learning Studio Instance, so storing it in Azure Storage allows users to use the data anytime they like. \n",
475
483
"Azure Storage is comprised of three parts: accounts, containers, and blobs, as seen in the image below. Accounts contain containers which act as folders. Containers hold files, aka blobs and folders or subfolders.\n",
"When you create the Azure Machine Learning service, it will automatically create a storage account under the name of the workspace you created before. This allows you to either use the previously made storage account or create a new one. Instructions for both options are outlined below.\n",
480
488
"**Option 1:Premade Storage Accounts**\n",
@@ -485,7 +493,8 @@
485
493
"5. The console will list all the containers and files created as shown in the figure below. Some of these containers and files are automatically created when we create our Machine Learning Service:\n",
0 commit comments