edegan.com - User contributions [en]

Mechanical Turk (Tool)

2016-07-09T20:21:43Z

DanFehder: /* Creating a New Project Example */

{{McNair Projects
|Project Title=Mechanical Turk (Tool),
|Topic Area=Entrepreneurship Ecosystems,
|Owner=Daniel Fehder,
|Start Term=Summer 2016,
|Status=Active,
}}
==Description==
The purpose of this page is to introduce people to the use of mechanical turk in data processing. The document is structured as follows: 1. It begins by describing the mechanical turk and the many ways in which it can be used. 2. it provides simple getting started instructions that allows a new user to access the mechanical turk system and begin a new project. 3. We give an example of a project with sample code.

==What is Mechanical Turk==
Mechanical Turk (Mturk) is a system that allows people to outsource work to many different people in an efficient way. For the purposes of the McNair center, we will be focused on the use of Mturk for the acquisition and cleaning of data. This is a great way to look up or clean data when you have a small number of easily understood steps that need to be repeated many times. If you data task fits this definition, then it is worth thinking about turning it into a Mturk task. In the example below, we think about how to find all the Twitter handles for a set of companies in a spreadsheet. If you were to do this by hand as an RA, you would start with the spreadsheet and go through each row searching on either google or twitter for each company. In Mturk, you would create a project. In that project, you would create a task template that would provide a set of overall instructions as well as hooks to fill in specific information about one row from your spreadsheet. When the turker receives their assignment, or HIT, they will see both the overall instructions and the specific information for that row of data in your spreadsheet. The Mturk system allows many people to work on your spreadsheet in parallel allowing the work to be completed much more quickly. If this is confusing, we will provide a concrete example below. For now, just bear in mind some essential vocabulary.

;Mechanical Turk Vocabulary
: ''Requester:'' the people posting work on the system
: ''HIT:'' one task completed by a worker
: ''Project:'' a collection of HITs
: ''Turker:'' a worker on Mturk

==Accessing the Mechanical Turk Platform==
* go to [https://requester.mturk.com/ The Mechanical Turk Requester page]
* Log into the system using the following{{clear}}
: email: esi@rice.edu
: pass: 9Million!

* To create a new project, click on the Create link and follow the directions in the Create Project Example section below
* To modify an existing project, follow the directions in the Modify an Existing Project section below

==Creating a New Project Example==
In the steps below, we describe the creation of a Turk project that asks Turk workers to find the twitter handles of companies. It will take as input a series of google search queries in csv form and the workers to enter the search strings into google and look to see if there are google handles that are returned on the first page of the search results.

'''Step 1, Project Info:''' Once you click on the create link, you will be brought to an interface with a number of text entry boxes. You want to summarize your project in ways that will be informative for the team as well as potential Turk workers choosing between projects. In the figure below, we describe a HIT Project FINISH.

Figure 1: Twitter Project Info{{clear}}
[[File:ProjectDescription.png]]

'''Step 2, Choosing Pay Level:''' Once you have named the project, you have to decide on pay scale (Reward per assignment) and the number of people working on each project (number of assignments per HIT). The higher the pay per HIT, the quicker your work will be completed by turkers, but you obviously do not want to waste money. A good rule of thumb is to work on the tasks you need completed by turkers for 30-60 minutes and then see how many rows you completed. We want the per HIT pay rate to roughly equal $6.00 - $10.00 in hourly wage to get things done efficiently on the system. If you decide to have more than 1 worker per HIT, it will be because you believe that the data task requires a certain amount of human judgement and you want to make sure that you only accept results that have been "verified" by multiple people. The last three parameters in this box determine how each HIT will be completed by each worker and how long the HIT stays in the system. You generally want "Time Allotted" to be 1 day. Expiration of the HIT doesn't matter that much. One of the last important choice in this screen is the "Auto-approve" option. The quicker the auto approve, the more likely that Turkers will take your task. For now, set it to 24 hours, but remember that you are responsible for regularly auditing results when you have a project up on the Turk system.

Figure 2: Cost Parameters in Mturk{{clear}}
[[File:Costing.png]]

'''Step 3, Design Layout:''' At this point, you have to design what the turker sees when they receive your assignment. While it is possible that one turker will complete multiple HITs, it is important to design the HIT so that it can be easily completed the first (and possibly only) time by the worker. In figure 3, below, you can see the initial design layout of the default data acquisition project in the turk system. It is an example HIT that asks turkers to find the website of a restaurant. Please note that this is not a great HIT in terms of the level of clarity of the instructions. We will provide guidelines on creating instructions below. For now, just notice a few features of the HIT. To the right of "Restaurant Name", there is a field called ${name}. This is actually a hook, or a blanks space, that will be populated with the actual name of a restaurant that will come from a spread sheet that you will upload into the turk system. Each HIT will correspond to one row of the spreadsheet. This is the same for the "Address" and "Phone Number" rows. The last key thing to notice is the "Website Address" field with a text entry box right below it. When a turker receives this HIT, they will paste the web address into this text box and you will receive a new spreadsheet with whatever they (add all the other turkers) pasted in the same row as the data you used to populate each HIT.

Now, How do you modify this HIT to reflect your actual data task? You can actually change the wording of the task directly in the editor screen. Make sure that all of the data element hooks (like ${name}) correspond to the actual names of the columns in the CSV file that you will upload on the turk system. But what if you need the task to look substantially different from the one you are looking at? If you click on the "Source", it will show you the actual html code of your HIT task as displayed in Figure 4. The turk system allows you to display a full website essentially for your HIT task with javascript, CSS, etc. As we develop our system at the McNair center, your will have more existing tasks to choose from, but when you need to actually build your own, some useful HTML references are listed below. When you have completed editing your HIT template, click on the "Save" Button and then move to "Preview". In this last screen, it will show you exactly what the turkers will see (Figure 5). If it looks correct, click "Finish"

;HTML references
: https://developer.mozilla.org/en-US/docs/Web/HTML/Element
: http://www.w3schools.com/html/default.asp

Figure 3: Design Layout {{clear}}
[[File:DesignLayout2.png]]

Figure 4: Design Layout, Raw HTML {{clear}}
[[File: DesignLayoutRaw.png]]

Figure 5: HIT Preview {{clear}}
[[File: HITPreview.png]]

'''Step 4, Publishing a Batch:''' Now that you have your HIT Template ready, you can post to the Turk system. To do so, Click the "Publish Batch" button in figure 6. It will then ask you to upload a csv file for that batch. Remember that this CSV file has to have headers that correspond to the data hooks in your template. Once you have successfully uploaded, you should be able to confirm payment and publish the batch. If there is not enough money preloaded on the turk system. Notify either Dan or Ed.

Figure 6: HIT Publishing{{clear}}
[[File: PublishBatch.png]]

==Existing HIT Library==
create a list of existing hits and what they do

==TDL with HITS==
* Data validation using javascript
*

Mechanical Turk (Tool)

2016-07-09T20:06:42Z

DanFehder: /* Creating a New Project Example */

{{McNair Projects
|Project Title=Mechanical Turk (Tool),
|Topic Area=Entrepreneurship Ecosystems,
|Owner=Daniel Fehder,
|Start Term=Summer 2016,
|Status=Active,
}}
==Description==
The purpose of this page is to introduce people to the use of mechanical turk in data processing. The document is structured as follows: 1. It begins by describing the mechanical turk and the many ways in which it can be used. 2. it provides simple getting started instructions that allows a new user to access the mechanical turk system and begin a new project. 3. We give an example of a project with sample code.

==What is Mechanical Turk==
Mechanical Turk (Mturk) is a system that allows people to outsource work to many different people in an efficient way. For the purposes of the McNair center, we will be focused on the use of Mturk for the acquisition and cleaning of data. This is a great way to look up or clean data when you have a small number of easily understood steps that need to be repeated many times. If you data task fits this definition, then it is worth thinking about turning it into a Mturk task. In the example below, we think about how to find all the Twitter handles for a set of companies in a spreadsheet. If you were to do this by hand as an RA, you would start with the spreadsheet and go through each row searching on either google or twitter for each company. In Mturk, you would create a project. In that project, you would create a task template that would provide a set of overall instructions as well as hooks to fill in specific information about one row from your spreadsheet. When the turker receives their assignment, or HIT, they will see both the overall instructions and the specific information for that row of data in your spreadsheet. The Mturk system allows many people to work on your spreadsheet in parallel allowing the work to be completed much more quickly. If this is confusing, we will provide a concrete example below. For now, just bear in mind some essential vocabulary.

;Mechanical Turk Vocabulary
: ''Requester:'' the people posting work on the system
: ''HIT:'' one task completed by a worker
: ''Project:'' a collection of HITs
: ''Turker:'' a worker on Mturk

==Accessing the Mechanical Turk Platform==
* go to [https://requester.mturk.com/ The Mechanical Turk Requester page]
* Log into the system using the following{{clear}}
: email: esi@rice.edu
: pass: 9Million!

* To create a new project, click on the Create link and follow the directions in the Create Project Example section below
* To modify an existing project, follow the directions in the Modify an Existing Project section below

==Creating a New Project Example==
In the steps below, we describe the creation of a Turk project that asks Turk workers to find the twitter handles of companies. It will take as input a series of google search queries in csv form and the workers to enter the search strings into google and look to see if there are google handles that are returned on the first page of the search results.

'''Step 1, Project Info:''' Once you click on the create link, you will be brought to an interface with a number of text entry boxes. You want to summarize your project in ways that will be informative for the team as well as potential Turk workers choosing between projects. In the figure below, we describe a HIT Project FINISH.

Figure 1: Twitter Project Info{{clear}}
[[File:ProjectDescription.png]]

'''Step 2, Choosing Pay Level:''' Once you have named the project, you have to decide on pay scale (Reward per assignment) and the number of people working on each project (number of assignments per HIT). The higher the pay per HIT, the quicker your work will be completed by turkers, but you obviously do not want to waste money. A good rule of thumb is to work on the tasks you need completed by turkers for 30-60 minutes and then see how many rows you completed. We want the per HIT pay rate to roughly equal $6.00 - $10.00 in hourly wage to get things done efficiently on the system. If you decide to have more than 1 worker per HIT, it will be because you believe that the data task requires a certain amount of human judgement and you want to make sure that you only accept results that have been "verified" by multiple people. The last three parameters in this box determine how each HIT will be completed by each worker and how long the HIT stays in the system. You generally want "Time Allotted" to be 1 day. Expiration of the HIT doesn't matter that much. One of the last important choice in this screen is the "Auto-approve" option. The quicker the auto approve, the more likely that Turkers will take your task. For now, set it to 24 hours, but remember that you are responsible for regularly auditing results when you have a project up on the Turk system.

Figure 2: Cost Parameters in Mturk{{clear}}
[[File:Costing.png]]

'''Step 3, Design Layout:''' At this point, you have to design what the turker sees when they receive your assignment. While it is possible that one turker will complete multiple HITs, it is important to design the HIT so that it can be easily completed the first (and possibly only) time by the worker. In figure 3, below, you can see the initial design layout of the default data acquisition project in the turk system. It is an example HIT that asks turkers to find the website of a restaurant. Please note that this is not a great HIT in terms of the level of clarity of the instructions. We will provide guidelines on creating instructions below. For now, just notice a few features of the HIT. To the right of "Restaurant Name", there is a field called ${name}. This is actually a hook, or a blanks space, that will be populated with the actual name of a restaurant that will come from a spread sheet that you will upload into the turk system. Each HIT will correspond to one row of the spreadsheet. This is the same for the "Address" and "Phone Number" rows. The last key thing to notice is the "Website Address" field with a text entry box right below it. When a turker receives this HIT, they will paste the web address into this text box and you will receive a new spreadsheet with whatever they (add all the other turkers) pasted in the same row as the data you used to populate each HIT.

Now, How do you modify this HIT to reflect your actual data task? You can actually change the wording of the task directly in the editor screen. Make sure that all of the data element hooks (like ${name}) correspond to the actual names of the columns in the CSV file that you will upload on the turk system. But what if you need the task to look substantially different from the one you are looking at? If you click on the "Source", it will show you the actual html code of your HIT task as displayed in Figure 4. The turk system allows you to display a full website essentially for your HIT task with javascript, CSS, etc. As we develop our system at the McNair center, your will have more existing tasks to choose from, but when you need to actually build your own, some useful HTML references are listed below. When you have completed editing your HIT template, click on the "Save" Button and then move to "Preview". In this last screen, it will show you exactly what the turkers will see (Figure 5). If it looks correct, click "Finish"

;HTML references
: https://developer.mozilla.org/en-US/docs/Web/HTML/Element
: REF 2

Figure 3: Design Layout {{clear}}
[[File:DesignLayout2.png]]

Figure 4: Design Layout, Raw HTML {{clear}}
[[File: DesignLayoutRaw.png]]

Figure 5: HIT Preview {{clear}}
[[File: HITPreview.png]]

'''Step 4, Publishing a Batch:''' Now that you have your HIT Template ready, you can post to the Turk system. To do so, Click the "Publish Batch" button in figure 6. It will then ask you to upload a csv file for that batch. Remember that this CSV file has to have headers that correspond to the data hooks in your template. Once you have successfully uploaded, you should be able to confirm payment and publish the batch. If there is not enough money preloaded on the turk system. Notify either Dan or Ed.

Figure 6: HIT Publishing{{clear}}
[[File: PublishBatch.png]]

==Existing HIT Library==
create a list of existing hits and what they do

==TDL with HITS==
* Data validation using javascript
*

Mechanical Turk (Tool)

2016-07-08T15:57:50Z

DanFehder:

{{McNair Projects
|Project Title=Mechanical Turk (Tool),
|Topic Area=Entrepreneurship Ecosystems,
|Owner=Daniel Fehder,
|Start Term=Summer 2016,
|Status=Active,
}}
==Description==
The purpose of this page is to introduce people to the use of mechanical turk in data processing. The document is structured as follows: 1. It begins by describing the mechanical turk and the many ways in which it can be used. 2. it provides simple getting started instructions that allows a new user to access the mechanical turk system and begin a new project. 3. We give an example of a project with sample code.

==What is Mechanical Turk==
Mechanical Turk (Mturk) is a system that allows people to outsource work to many different people in an efficient way. For the purposes of the McNair center, we will be focused on the use of Mturk for the acquisition and cleaning of data. This is a great way to look up or clean data when you have a small number of easily understood steps that need to be repeated many times. If you data task fits this definition, then it is worth thinking about turning it into a Mturk task. In the example below, we think about how to find all the Twitter handles for a set of companies in a spreadsheet. If you were to do this by hand as an RA, you would start with the spreadsheet and go through each row searching on either google or twitter for each company. In Mturk, you would create a project. In that project, you would create a task template that would provide a set of overall instructions as well as hooks to fill in specific information about one row from your spreadsheet. When the turker receives their assignment, or HIT, they will see both the overall instructions and the specific information for that row of data in your spreadsheet. The Mturk system allows many people to work on your spreadsheet in parallel allowing the work to be completed much more quickly. If this is confusing, we will provide a concrete example below. For now, just bear in mind some essential vocabulary.

;Mechanical Turk Vocabulary
: ''Requester:'' the people posting work on the system
: ''HIT:'' one task completed by a worker
: ''Project:'' a collection of HITs
: ''Turker:'' a worker on Mturk

==Accessing the Mechanical Turk Platform==
* go to [https://requester.mturk.com/ The Mechanical Turk Requester page]
* Log into the system using the following{{clear}}
: email: esi@rice.edu
: pass: 9Million!

* To create a new project, click on the Create link and follow the directions in the Create Project Example section below
* To modify an existing project, follow the directions in the Modify an Existing Project section below

==Creating a New Project Example==
In the steps below, we describe the creation of a Turk project that asks Turk workers to find the twitter handles of companies. It will take as input a series of google search queries in csv form and the workers to enter the search strings into google and look to see if there are google handles that are returned on the first page of the search results.

'''Step 1, Project Info:''' Once you click on the create link, you will be brought to an interface with a number of text entry boxes. You want to summarize your project in ways that will be informative for the team as well as potential Turk workers choosing between projects. In the figure below, we describe a HIT Project FINISH.

Figure 1: Twitter Project Info{{clear}}
[[File:ProjectDescription.png]]

'''Step 2, Choosing Pay Level:''' Once you have named the project, you have to decide on pay scale (Reward per assignment) and the number of people working on each project (number of assignments per HIT). The higher the pay per HIT, the quicker your work will be completed by turkers, but you obviously do not want to waste money. A good rule of thumb is to work on the tasks you need completed by turkers for 30-60 minutes and then see how many rows you completed. We want the per HIT pay rate to roughly equal $6.00 - $10.00 in hourly wage to get things done efficiently on the system. If you decide to have more than 1 worker per HIT, it will be because you believe that the data task requires a certain amount of human judgement and you want to make sure that you only accept results that have been "verified" by multiple people. The last three parameters in this box determine how each HIT will be completed by each worker and how long the HIT stays in the system. You generally want "Time Allotted" to be 1 day. Expiration of the HIT doesn't matter that much. One of the last important choice in this screen is the "Auto-approve" option. The quicker the auto approve, the more likely that Turkers will take your task. For now, set it to 24 hours, but remember that you are responsible for regularly auditing results when you have a project up on the Turk system.

Figure 2: Cost Parameters in Mturk{{clear}}
[[File:Costing.png]]

'''Step 3, Design Layout:''' At this point, you have to design what the turker sees when they receive your assignment. While it is possible that one turker will complete multiple HITs, it is important to design the HIT so that it can be easily completed the first (and possibly only) time by the worker. In figure 3, below, you can see the initial design layout of the default data acquisition project in the turk system. It is an example HIT that asks turkers to find the website of a restaurant. Please note that this is not a great HIT in terms of the level of clarity of the instructions. We will provide guidelines on creating instructions below. For now, just notice a few features of the HIT. To the right of "Restaurant Name", there is a field called ${name}. This is actually a hook, or a blanks space, that will be populated with the actual name of a restaurant that will come from a spread sheet that you will upload into the turk system. Each HIT will correspond to one row of the spreadsheet. This is the same for the "Address" and "Phone Number" rows. The last key thing to notice is the "Website Address" field with a text entry box right below it. When a turker receives this HIT, they will paste the web address into this text box and you will receive a new spreadsheet with whatever they (add all the other turkers) pasted in the same row as the data you used to populate each HIT.

Now, How do you modify this HIT to reflect your actual data task? You can actually change the wording of the task directly in the editor screen. Make sure that all of the data element hooks (like ${name}) correspond to the actual names of the columns in the CSV file that you will upload on the turk system. But what if you need the task to look substantially different from the one you are looking at? If you click on the "Source", it will show you the actual html code of your HIT task as displayed in Figure 4. The turk system allows you to display a full website essentially for your HIT task with javascript, CSS, etc. As we develop our system at the McNair center, your will have more existing tasks to choose from, but when you need to actually build your own, some useful HTML references are listed below. When you have completed editing your HIT template, click on the "Save" Button and then move to "Preview". In this last screen, it will show you exactly what the turkers will see (Figure 5). If it looks correct, click "Finish"

;HTML references
: REF 1
: REF 2

Figure 3: Design Layout {{clear}}
[[File:DesignLayout2.png]]

Figure 4: Design Layout, Raw HTML {{clear}}
[[File: DesignLayoutRaw.png]]

Figure 5: HIT Preview {{clear}}
[[File: HITPreview.png]]

'''Step 4, Publishing a Batch:''' Now that you have your HIT Template ready, you can post to the Turk system. To do so, Click the "Publish Batch" button in figure 6. It will then ask you to upload a csv file for that batch. Remember that this CSV file has to have headers that correspond to the data hooks in your template. Once you have successfully uploaded, you should be able to confirm payment and publish the batch. If there is not enough money preloaded on the turk system. Notify either Dan or Ed.

Figure 6: HIT Publishing{{clear}}
[[File: PublishBatch.png]]

==Existing HIT Library==
create a list of existing hits and what they do

==TDL with HITS==
* Data validation using javascript
*

Mechanical Turk (Tool)

2016-07-08T15:57:15Z

DanFehder: /* Creating a New Project Example */

{{McNair Projects
|Project Title=Mechanical Turk (Tool),
|Topic Area=Entrepreneurship Ecosystems,
|Owner=Daniel Fehder,
|Start Term=Summer 2016,
|Status=Active,
}}
==Description==
The purpose of this page is to introduce people to the use of mechanical turk in data processing. The document is structured as follows: 1. It begins by describing the mechanical turk and the many ways in which it can be used. 2. it provides simple getting started instructions that allows a new user to access the mechanical turk system and begin a new project. 3. We give an example of a project with sample code.

==What is Mechanical Turk==
Mechanical Turk (Mturk) is a system that allows people to outsource work to many different people in an efficient way. For the purposes of the McNair center, we will be focused on the use of Mturk for the acquisition and cleaning of data. This is a great way to look up or clean data when you have a small number of easily understood steps that need to be repeated many times. If you data task fits this definition, then it is worth thinking about turning it into a Mturk task. In the example below, we think about how to find all the Twitter handles for a set of companies in a spreadsheet. If you were to do this by hand as an RA, you would start with the spreadsheet and go through each row searching on either google or twitter for each company. In Mturk, you would create a project. In that project, you would create a task template that would provide a set of overall instructions as well as hooks to fill in specific information about one row from your spreadsheet. When the turker receives their assignment, or HIT, they will see both the overall instructions and the specific information for that row of data in your spreadsheet. The Mturk system allows many people to work on your spreadsheet in parallel allowing the work to be completed much more quickly. If this is confusing, we will provide a concrete example below. For now, just bear in mind some essential vocabulary.

;Mechanical Turk Vocabulary
: ''Requester:'' the people posting work on the system
: ''HIT:'' one task completed by a worker
: ''Project:'' a collection of HITs
: ''Turker:'' a worker on Mturk

==Accessing the Mechanical Turk Platform==
* go to [https://requester.mturk.com/ The Mechanical Turk Requester page]
* Log into the system using the following{{clear}}
: email: esi@rice.edu
: pass: 9Million!

* To create a new project, click on the Create link and follow the directions in the Create Project Example section below
* To modify an existing project, follow the directions in the Modify an Existing Project section below

==Creating a New Project Example==
In the steps below, we describe the creation of a Turk project that asks Turk workers to find the twitter handles of companies. It will take as input a series of google search queries in csv form and the workers to enter the search strings into google and look to see if there are google handles that are returned on the first page of the search results.

'''Step 1, Project Info:''' Once you click on the create link, you will be brought to an interface with a number of text entry boxes. You want to summarize your project in ways that will be informative for the team as well as potential Turk workers choosing between projects. In the figure below, we describe a HIT Project FINISH.

Figure 1: Twitter Project Info{{clear}}
[[File:ProjectDescription.png]]

'''Step 2, Choosing Pay Level:''' Once you have named the project, you have to decide on pay scale (Reward per assignment) and the number of people working on each project (number of assignments per HIT). The higher the pay per HIT, the quicker your work will be completed by turkers, but you obviously do not want to waste money. A good rule of thumb is to work on the tasks you need completed by turkers for 30-60 minutes and then see how many rows you completed. We want the per HIT pay rate to roughly equal $6.00 - $10.00 in hourly wage to get things done efficiently on the system. If you decide to have more than 1 worker per HIT, it will be because you believe that the data task requires a certain amount of human judgement and you want to make sure that you only accept results that have been "verified" by multiple people. The last three parameters in this box determine how each HIT will be completed by each worker and how long the HIT stays in the system. You generally want "Time Allotted" to be 1 day. Expiration of the HIT doesn't matter that much. One of the last important choice in this screen is the "Auto-approve" option. The quicker the auto approve, the more likely that Turkers will take your task. For now, set it to 24 hours, but remember that you are responsible for regularly auditing results when you have a project up on the Turk system.

Figure 2: Cost Parameters in Mturk{{clear}}
[[File:Costing.png]]

'''Step 3, Design Layout:''' At this point, you have to design what the turker sees when they receive your assignment. While it is possible that one turker will complete multiple HITs, it is important to design the HIT so that it can be easily completed the first (and possibly only) time by the worker. In figure 3, below, you can see the initial design layout of the default data acquisition project in the turk system. It is an example HIT that asks turkers to find the website of a restaurant. Please note that this is not a great HIT in terms of the level of clarity of the instructions. We will provide guidelines on creating instructions below. For now, just notice a few features of the HIT. To the right of "Restaurant Name", there is a field called ${name}. This is actually a hook, or a blanks space, that will be populated with the actual name of a restaurant that will come from a spread sheet that you will upload into the turk system. Each HIT will correspond to one row of the spreadsheet. This is the same for the "Address" and "Phone Number" rows. The last key thing to notice is the "Website Address" field with a text entry box right below it. When a turker receives this HIT, they will paste the web address into this text box and you will receive a new spreadsheet with whatever they (add all the other turkers) pasted in the same row as the data you used to populate each HIT.

Now, How do you modify this HIT to reflect your actual data task? You can actually change the wording of the task directly in the editor screen. Make sure that all of the data element hooks (like ${name}) correspond to the actual names of the columns in the CSV file that you will upload on the turk system. But what if you need the task to look substantially different from the one you are looking at? If you click on the "Source", it will show you the actual html code of your HIT task as displayed in Figure 4. The turk system allows you to display a full website essentially for your HIT task with javascript, CSS, etc. As we develop our system at the McNair center, your will have more existing tasks to choose from, but when you need to actually build your own, some useful HTML references are listed below. When you have completed editing your HIT template, click on the "Save" Button and then move to "Preview". In this last screen, it will show you exactly what the turkers will see (Figure 5). If it looks correct, click "Finish"

;HTML references
: REF 1
: REF 2

Figure 3: Design Layout {{clear}}
[[File:DesignLayout2.png]]

Figure 4: Design Layout, Raw HTML {{clear}}
[[File: DesignLayoutRaw.png]]

Figure 5: HIT Preview {{clear}}
[[File: HITPreview.png]]

'''Step 4, Publishing a Batch:''' Now that you have your HIT Template ready, you can post to the Turk system. To do so, Click the "Publish Batch" button in figure 6. It will then ask you to upload a csv file for that batch. Remember that this CSV file has to have headers that correspond to the data hooks in your template. Once you have successfully uploaded, you should be able to confirm payment and publish the batch. If there is not enough money preloaded on the turk system. Notify either Dan or Ed.

Figure 6: HIT Publishing{{clear}}
[[File: PublishBatch.png]]

==Existing HIT Library==
create a list of existing hits and what they do

==TDL with HITS==
* Data validation using javascript
*

==Hash==
import requests
response = requests.get(
"https://www.eventbriteapi.com/v3/organizers/2300226659/events/",
headers = {
"Authorization": "Bearer CRAQ5MAXEGHKEXSUSWXN",
},
verify = True,
)

File:PublishBatch.png

2016-07-08T15:55:07Z

DanFehder:

Mechanical Turk (Tool)

2016-07-08T15:52:40Z

DanFehder: /* Creating a New Project Example */

{{McNair Projects
|Project Title=Mechanical Turk (Tool),
|Topic Area=Entrepreneurship Ecosystems,
|Owner=Daniel Fehder,
|Start Term=Summer 2016,
|Status=Active,
}}
==Description==
The purpose of this page is to introduce people to the use of mechanical turk in data processing. The document is structured as follows: 1. It begins by describing the mechanical turk and the many ways in which it can be used. 2. it provides simple getting started instructions that allows a new user to access the mechanical turk system and begin a new project. 3. We give an example of a project with sample code.

==What is Mechanical Turk==
Mechanical Turk (Mturk) is a system that allows people to outsource work to many different people in an efficient way. For the purposes of the McNair center, we will be focused on the use of Mturk for the acquisition and cleaning of data. This is a great way to look up or clean data when you have a small number of easily understood steps that need to be repeated many times. If you data task fits this definition, then it is worth thinking about turning it into a Mturk task. In the example below, we think about how to find all the Twitter handles for a set of companies in a spreadsheet. If you were to do this by hand as an RA, you would start with the spreadsheet and go through each row searching on either google or twitter for each company. In Mturk, you would create a project. In that project, you would create a task template that would provide a set of overall instructions as well as hooks to fill in specific information about one row from your spreadsheet. When the turker receives their assignment, or HIT, they will see both the overall instructions and the specific information for that row of data in your spreadsheet. The Mturk system allows many people to work on your spreadsheet in parallel allowing the work to be completed much more quickly. If this is confusing, we will provide a concrete example below. For now, just bear in mind some essential vocabulary.

;Mechanical Turk Vocabulary
: ''Requester:'' the people posting work on the system
: ''HIT:'' one task completed by a worker
: ''Project:'' a collection of HITs
: ''Turker:'' a worker on Mturk

==Accessing the Mechanical Turk Platform==
* go to [https://requester.mturk.com/ The Mechanical Turk Requester page]
* Log into the system using the following{{clear}}
: email: esi@rice.edu
: pass: 9Million!

* To create a new project, click on the Create link and follow the directions in the Create Project Example section below
* To modify an existing project, follow the directions in the Modify an Existing Project section below

==Creating a New Project Example==
In the steps below, we describe the creation of a Turk project that asks Turk workers to find the twitter handles of companies. It will take as input a series of google search queries in csv form and the workers to enter the search strings into google and look to see if there are google handles that are returned on the first page of the search results.

'''Step 1, Project Info:''' Once you click on the create link, you will be brought to an interface with a number of text entry boxes. You want to summarize your project in ways that will be informative for the team as well as potential Turk workers choosing between projects. In the figure below, we describe a HIT Project FINISH.

Figure 1: Twitter Project Info{{clear}}
[[File:ProjectDescription.png]]

'''Step 2, Choosing Pay Level:''' Once you have named the project, you have to decide on pay scale (Reward per assignment) and the number of people working on each project (number of assignments per HIT). The higher the pay per HIT, the quicker your work will be completed by turkers, but you obviously do not want to waste money. A good rule of thumb is to work on the tasks you need completed by turkers for 30-60 minutes and then see how many rows you completed. We want the per HIT pay rate to roughly equal $6.00 - $10.00 in hourly wage to get things done efficiently on the system. If you decide to have more than 1 worker per HIT, it will be because you believe that the data task requires a certain amount of human judgement and you want to make sure that you only accept results that have been "verified" by multiple people. The last three parameters in this box determine how each HIT will be completed by each worker and how long the HIT stays in the system. You generally want "Time Allotted" to be 1 day. Expiration of the HIT doesn't matter that much. One of the last important choice in this screen is the "Auto-approve" option. The quicker the auto approve, the more likely that Turkers will take your task. For now, set it to 24 hours, but remember that you are responsible for regularly auditing results when you have a project up on the Turk system.

Figure 2: Cost Parameters in Mturk{{clear}}
[[File:Costing.png]]

'''Step 3, Design Layout:''' At this point, you have to design what the turker sees when they receive your assignment. While it is possible that one turker will complete multiple HITs, it is important to design the HIT so that it can be easily completed the first (and possibly only) time by the worker. In figure 3, below, you can see the initial design layout of the default data acquisition project in the turk system. It is an example HIT that asks turkers to find the website of a restaurant. Please note that this is not a great HIT in terms of the level of clarity of the instructions. We will provide guidelines on creating instructions below. For now, just notice a few features of the HIT. To the right of "Restaurant Name", there is a field called ${name}. This is actually a hook, or a blanks space, that will be populated with the actual name of a restaurant that will come from a spread sheet that you will upload into the turk system. Each HIT will correspond to one row of the spreadsheet. This is the same for the "Address" and "Phone Number" rows. The last key thing to notice is the "Website Address" field with a text entry box right below it. When a turker receives this HIT, they will paste the web address into this text box and you will receive a new spreadsheet with whatever they (add all the other turkers) pasted in the same row as the data you used to populate each HIT.

Now, How do you modify this HIT to reflect your actual data task? You can actually change the wording of the task directly in the editor screen. Make sure that all of the data element hooks (like ${name}) correspond to the actual names of the columns in the CSV file that you will upload on the turk system. But what if you need the task to look substantially different from the one you are looking at? If you click on the "Source", it will show you the actual html code of your HIT task as displayed in Figure 4. The turk system allows you to display a full website essentially for your HIT task with javascript, CSS, etc. As we develop our system at the McNair center, your will have more existing tasks to choose from, but when you need to actually build your own, some useful HTML references are listed below. When you have completed editing your HIT template, click on the "Save" Button and then move to "Preview". In this last screen, it will show you exactly what the turkers will see (Figure 5). If it looks correct, click "Finish"

;HTML references
: REF 1
: REF 2

Figure 3: Design Layout {{clear}}
[[File:DesignLayout2.png]]

Figure 4: Design Layout, Raw HTML {{clear}}
[[File: DesignLayoutRaw.png]]

Figure 5: HIT Preview {{clear}}
[[File: HITPreview.png]]

==Existing HIT Library==
create a list of existing hits and what they do

==TDL with HITS==
* Data validation using javascript
*

==Hash==
import requests
response = requests.get(
"https://www.eventbriteapi.com/v3/organizers/2300226659/events/",
headers = {
"Authorization": "Bearer CRAQ5MAXEGHKEXSUSWXN",
},
verify = True,
)

File:HITPreview.png

2016-07-08T15:52:22Z

DanFehder:

File:DesignLayout2.png

2016-07-08T15:49:25Z

DanFehder:

Mechanical Turk (Tool)

2016-07-08T15:45:41Z

DanFehder: /* Creating a New Project Example */

{{McNair Projects
|Project Title=Mechanical Turk (Tool),
|Topic Area=Entrepreneurship Ecosystems,
|Owner=Daniel Fehder,
|Start Term=Summer 2016,
|Status=Active,
}}
==Description==
The purpose of this page is to introduce people to the use of mechanical turk in data processing. The document is structured as follows: 1. It begins by describing the mechanical turk and the many ways in which it can be used. 2. it provides simple getting started instructions that allows a new user to access the mechanical turk system and begin a new project. 3. We give an example of a project with sample code.

==What is Mechanical Turk==
Mechanical Turk (Mturk) is a system that allows people to outsource work to many different people in an efficient way. For the purposes of the McNair center, we will be focused on the use of Mturk for the acquisition and cleaning of data. This is a great way to look up or clean data when you have a small number of easily understood steps that need to be repeated many times. If you data task fits this definition, then it is worth thinking about turning it into a Mturk task. In the example below, we think about how to find all the Twitter handles for a set of companies in a spreadsheet. If you were to do this by hand as an RA, you would start with the spreadsheet and go through each row searching on either google or twitter for each company. In Mturk, you would create a project. In that project, you would create a task template that would provide a set of overall instructions as well as hooks to fill in specific information about one row from your spreadsheet. When the turker receives their assignment, or HIT, they will see both the overall instructions and the specific information for that row of data in your spreadsheet. The Mturk system allows many people to work on your spreadsheet in parallel allowing the work to be completed much more quickly. If this is confusing, we will provide a concrete example below. For now, just bear in mind some essential vocabulary.

;Mechanical Turk Vocabulary
: ''Requester:'' the people posting work on the system
: ''HIT:'' one task completed by a worker
: ''Project:'' a collection of HITs
: ''Turker:'' a worker on Mturk

==Accessing the Mechanical Turk Platform==
* go to [https://requester.mturk.com/ The Mechanical Turk Requester page]
* Log into the system using the following{{clear}}
: email: esi@rice.edu
: pass: 9Million!

* To create a new project, click on the Create link and follow the directions in the Create Project Example section below
* To modify an existing project, follow the directions in the Modify an Existing Project section below

==Creating a New Project Example==
In the steps below, we describe the creation of a Turk project that asks Turk workers to find the twitter handles of companies. It will take as input a series of google search queries in csv form and the workers to enter the search strings into google and look to see if there are google handles that are returned on the first page of the search results.

'''Step 1, Project Info:''' Once you click on the create link, you will be brought to an interface with a number of text entry boxes. You want to summarize your project in ways that will be informative for the team as well as potential Turk workers choosing between projects. In the figure below, we describe a HIT Project FINISH.

Figure 1: Twitter Project Info{{clear}}
[[File:ProjectDescription.png]]

'''Step 2, Choosing Pay Level:''' Once you have named the project, you have to decide on pay scale (Reward per assignment) and the number of people working on each project (number of assignments per HIT). The higher the pay per HIT, the quicker your work will be completed by turkers, but you obviously do not want to waste money. A good rule of thumb is to work on the tasks you need completed by turkers for 30-60 minutes and then see how many rows you completed. We want the per HIT pay rate to roughly equal $6.00 - $10.00 in hourly wage to get things done efficiently on the system. If you decide to have more than 1 worker per HIT, it will be because you believe that the data task requires a certain amount of human judgement and you want to make sure that you only accept results that have been "verified" by multiple people. The last three parameters in this box determine how each HIT will be completed by each worker and how long the HIT stays in the system. You generally want "Time Allotted" to be 1 day. Expiration of the HIT doesn't matter that much. One of the last important choice in this screen is the "Auto-approve" option. The quicker the auto approve, the more likely that Turkers will take your task. For now, set it to 24 hours, but remember that you are responsible for regularly auditing results when you have a project up on the Turk system.

Figure 2: Cost Parameters in Mturk{{clear}}
[[File:Costing.png]]

'''Step 3, Design Layout:''' At this point, you have to design what the turker sees when they receive your assignment. While it is possible that one turker will complete multiple HITs, it is important to design the HIT so that it can be easily completed the first (and possibly only) time by the worker. In figure 3, below, you can see the initial design layout of the default data acquisition project in the turk system. It is an example HIT that asks turkers to find the website of a restaurant. Please note that this is not a great HIT in terms of the level of clarity of the instructions. We will provide guidelines on creating instructions below. For now, just notice a few features of the HIT. To the right of "Restaurant Name", there is a field called ${name}. This is actually a hook, or a blanks space, that will be populated with the actual name of a restaurant that will come from a spread sheet that you will upload into the turk system. Each HIT will correspond to one row of the spreadsheet. This is the same for the "Address" and "Phone Number" rows. The last key thing to notice is the "Website Address" field with a text entry box right below it. When a turker receives this HIT, they will paste the web address into this text box and you will receive a new spreadsheet with whatever they (add all the other turkers) pasted in the same row as the data you used to populate each HIT.

Now, How do you modify this HIT to reflect your actual data task? You can actually change the wording of the task directly in the editor screen. Make sure that all of the data element hooks (like ${name}) correspond to the actual names of the columns in the CSV file that you will upload on the turk system. But what if you need the task to look substantially different from the one you are looking at? If you click on the "Source", it will show you the actual html code of your HIT task as displayed in Figure 4. The turk system allows you to display a full website essentially for your HIT task with javascript, CSS, etc. As we develop our system at the McNair center, your will have more existing tasks to choose from, but when you need to actually build your own, some useful HTML references are listed below.

;HTML references
: REF 1
: REF 2

Figure 3: Design Layout {{clear}}
[[File:DesignLayout.png]]

Figure 4: Design Layout, Raw HTML {{clear}}
[[File: DesignLayoutRaw.png]]

==Existing HIT Library==
create a list of existing hits and what they do

==TDL with HITS==
* Data validation using javascript
*

==Hash==
import requests
response = requests.get(
"https://www.eventbriteapi.com/v3/organizers/2300226659/events/",
headers = {
"Authorization": "Bearer CRAQ5MAXEGHKEXSUSWXN",
},
verify = True,
)

File:DesignLayoutRaw.png

2016-07-08T15:43:26Z

DanFehder:

Mechanical Turk (Tool)

{{McNair Projects
|Project Title=Mechanical Turk (Tool),
|Topic Area=Entrepreneurship Ecosystems,
|Owner=Daniel Fehder,
|Start Term=Summer 2016,
|Status=Active,
}}
==Description==
'''Notes''':

Accelerator Industry Classification (Data)

2016-06-21T14:08:07Z

DanFehder: /* Introduction and steps */

{{McNair Projects
|Project Title=Accelerator Industry Classification (Data),
|Topic Area=Entrepreneurship Ecosystems,
|Owner=Veeral Shah,
|Start Term=Summer 2016,
|Keywords=Mechanical Turk,
|Primary Billing=AccNBER01,
}}
Accelerators classified by the industry of which the accelerators' cohorts typically reside in.

==Introduction and steps==

The purpose of this project is to be able to give a NAICS code to each startup in our sample of startups that have attended accelerators. This will allow us to determine the specialization (if any) of each of the startup accelerators in our sample. The first few steps of the process are meant to get you up to speed.

'''Step 1: Retrieve URLs for each of the companies'''

For the first task, we will get the URLs for the company websites of every company that entered an accelerator. To do so, we will use “Porfolio Firms” Tab of the Google Sheet called “Accelerator-Portfolio-NAICS”.

The link to the google sheet is here: https://docs.google.com/spreadsheets/d/1Bw3qX7xLs5lSv577pS9YUtURDofYdqOza2MuP4dbthw/edit?usp=sharing

Here are the steps, each refers step refers to this sheet explicitely:

1. Look for a row that does not have the URL column filled out

2. Check to make sure that no one else is working on that row by seeing if someone has placed their cursor on that row

3. Now put your cursor on that row by clicking on the row’s Name column (this will make sure no one duplicates your work)

4. Now copy the Accelerator and Name columns of that row

5. Paste them into the google search box and search

6. Find a website that references both the accelerator name and the startup name

7. Click on that website’s link in google

8. See if the webpage includes a URL for the startup

9. Copy the company’s URL to the URL column in the row of the startup in the “Accelerator-Portfolio-NAICS” sheet

10. Put your initials (first, middle and last) in the “URL-ENTERER” column of the spreadsheet

11. If you have time, go back to step 1 and repeat

'''Step 2: Measure your productivity at Step 1'''

1. Repeat Step 1 for 1.5 - 2 hours while keeping track of how many rows you have completed (I do it by noting start and end row throughout this period). For this period of work, really try to focus and see how well you can do. We are trying to establish a top end of the productivity possible when running this task in this way.

2. Make note of your production for that time period both overall and in terms of per minute completion rate in the lab notes section below.

3. In addition to the simple fact of how fast you were able to complete rows of data, please make notes of what took the most amount of time on average for each step. Essentially, I want you to do a rough version of a "time motion" study (feel free to look that up) that will reveal potential sources of productivity gain if we could automate them.

'''Step 3: Install a new piece of software into the server system'''

1. Find time with Ed to install the software provided at the links below onto the system

https://www.dropbox.com/s/egau7yuho3wkjlo/gscript.py?dl=0

https://www.dropbox.com/s/afqzcpsxvvl1k2n/glink.py?dl=0

1a. While you wait for Ed to help you with the installation, I want you to read through the code in these two programs. I want you to really puzzle through how this all works until you think you understand it. As you do this, I want you to keep good notes on the wiki (including potentially screenshots) under the lab notebook tab. Also, I want you to freely message me with questions as you go.

'''Step 4: Run the new software on a limited batch of the spreadsheet above'''

1. Call me before you do this, but this step should be self explanatory

2. The program will output the same sheet you inputted but with appended search results. This should make it simpler to find a URL for the website

3. Create a set of steps that use the output of the program to find company URLS and note those steps in your lab notes

==Lab Notes==

==Women==

==International==

Accelerator Industry Classification (Data)

2016-06-21T13:55:41Z

DanFehder: /* Energy/Environmental */

Accelerator Industry Classification (Data)

2016-06-21T13:55:17Z

DanFehder: /* Introduction and steps */

Accelerator Industry Classification (Data)

2016-06-20T16:53:20Z

DanFehder: /* Introduction and steps */